From f5e49b158bb820ca99d64850a8f1f5a0c7f4eb47 Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Wed, 28 Jul 2021 12:17:13 +0200 Subject: [PATCH] rewrite storage-driver section - Reduce use of "Community Edition", and some sections that were written for Docker Enterprise. - Add notes about deprecation of AuFS in the overview. - Update the storage driver index page to use BuildKit, and rephrase some bits that made assumptions about each Dockerfile instruction producing a new layer. - Updated some examples to be usable on Docker Desktop, which does not provide direct access to the daemon's storage directoreis. - Add some notes linking to relevant sections about formatting output. Signed-off-by: Sebastiaan van Stijn --- storage/storagedriver/index.md | 409 ++++++++++++------ .../storagedriver/select-storage-driver.md | 187 ++++---- 2 files changed, 342 insertions(+), 254 deletions(-) diff --git a/storage/storagedriver/index.md b/storage/storagedriver/index.md index 90df885614..45e75eb3d8 100644 --- a/storage/storagedriver/index.md +++ b/storage/storagedriver/index.md @@ -14,14 +14,22 @@ stores images, and how these images are used by containers. You can use this information to make informed choices about the best way to persist data from your applications and avoid performance problems along the way. -Storage drivers allow you to create data in the writable layer of your container. -The files won't be persisted after the container is deleted, and both read and -write speeds are lower than native file system performance. +## Storage drivers versus Docker volumes - > **Note**: Operations that are known to be problematic include write-intensive database storage, -particularly when pre-existing data exists in the read-only layer. More details are provided in this document. +Docker uses storage drivers to store image layers, and to store data in the +writable layer of a container. The container's writable layer does not persist +after the container is deleted, but is suitable for storing ephemeral data that +is generated at runtime. Storage drivers are optimized for space efficiency, but +(depending on the storage driver) write speeds are lower than native file system +performance, especially for storage drivers that a use copy-on-write filesystem. +Write-intensive applications, such as database storage, are impacted by a +performance overhead, particularly if pre-existing data exists in the read-only +layer. -[Learn how to use volumes](../volumes.md) to persist data and improve performance. +Use Docker volumes for write-intensive data, data that must persist beyond the +container's lifespan, and data that must be shared between containers. Refer to +the [volumes section](../volumes.md) to learn how to use volumes to persist data +and improve performance. ## Images and layers @@ -32,24 +40,37 @@ read-only. Consider the following Dockerfile: ```dockerfile # syntax=docker/dockerfile:1 FROM ubuntu:18.04 +LABEL org.opencontainers.image.authors="org@example.com" COPY . /app RUN make /app +RUN rm -r $HOME/.cache CMD python /app/app.py ``` -This Dockerfile contains four commands, each of which creates a layer. The -`FROM` statement starts out by creating a layer from the `ubuntu:18.04` image. -The `COPY` command adds some files from your Docker client's current directory. -The `RUN` command builds your application using the `make` command. Finally, -the last layer specifies what command to run within the container. +This Dockerfile contains four commands. Commands that modify the filesystem create +a layer. The`FROM` statement starts out by creating a layer from the `ubuntu:18.04` +image. The `LABEL` command only modifies the image's metadata, and does not produce +a new layer. The `COPY` command adds some files from your Docker client's current +directory. The first `RUN` command builds your application using the `make` command, +and writes the result to a new layer. The second `RUN` command removes a cache +directory, and writes the result to a new layer. Finally, the `CMD` instruction +specifies what command to run within the container, which only modifies the +image's metadata, which does not produce an image layer. -Each layer is only a set of differences from the layer before it. The layers are -stacked on top of each other. When you create a new container, you add a new -writable layer on top of the underlying layers. This layer is often called the -"container layer". All changes made to the running container, such as writing -new files, modifying existing files, and deleting files, are written to this thin -writable container layer. The diagram below shows a container based on the Ubuntu -15.04 image. +Each layer is only a set of differences from the layer before it. Note that both +_adding_, and _removing_ files will result in a new layer. In the example above, +the `$HOME/.cache` directory is removed, but will still be available in the +previous layer and add up to the image's total size. Refer to the +[Best practices for writing Dockerfiles](../../develop/develop-images/dockerfile_best-practices.md) +and [use multi-stage builds](../../develop/develop-images/multistage-build.md) +sections to learn how to optimize your Dockerfiles for efficient images. + +The layers are stacked on top of each other. When you create a new container, +you add a new writable layer on top of the underlying layers. This layer is often +called the "container layer". All changes made to the running container, such as +writing new files, modifying existing files, and deleting files, are written to +this thin writable container layer. The diagram below shows a container based +on an `ubuntu:15.04` image. ![Layers of a container based on the Ubuntu image](images/container-layers.jpg) @@ -71,15 +92,18 @@ multiple containers sharing the same Ubuntu 15.04 image. ![Containers sharing same image](images/sharing-layers.jpg) -> **Note**: If you need multiple images to have shared access to the exact -> same data, store this data in a Docker volume and mount it into your -> containers. - Docker uses storage drivers to manage the contents of the image layers and the writable container layer. Each storage driver handles the implementation differently, but all drivers use stackable image layers and the copy-on-write (CoW) strategy. +> **Note** +> +> Use Docker volumes if you need multiple containers to have shared access to +> the exact same data. Refer to the [volumes section](../volumes.md) to learn +> about volumes. + + ## Container size on disk To view the approximate size of a running container, you can use the `docker ps -s` @@ -87,7 +111,6 @@ command. Two different columns relate to size. - `size`: the amount of data (on disk) that is used for the writable layer of each container. - - `virtual size`: the amount of data used for the read-only image data used by the container plus the container's writable layer `size`. Multiple containers may share some or all read-only @@ -106,9 +129,9 @@ these containers would be SUM (`size` of containers) plus one image size This also does not count the following additional ways a container can take up disk space: -- Disk space used for log files if you use the `json-file` logging driver. This - can be non-trivial if your container generates a large amount of logging data - and log rotation is not configured. +- Disk space used for log files stored by the [logging-driver](../../config/containers/logging/index.md). + This can be non-trivial if your container generates a large amount of logging + data and log rotation is not configured. - Volumes and bind mounts used by the container. - Disk space used for the container's configuration files, which are typically small. @@ -133,7 +156,7 @@ pulled down separately, and stored in Docker's local storage area, which is usually `/var/lib/docker/` on Linux hosts. You can see these layers being pulled in this example: -```bash +```console $ docker pull ubuntu:18.04 18.04: Pulling from library/ubuntu f476d66f5408: Pull complete @@ -149,7 +172,7 @@ local storage area. To examine the layers on the filesystem, list the contents of `/var/lib/docker/`. This example uses the `overlay2` storage driver: -```bash +```console $ ls /var/lib/docker/overlay2 16802227a96c24dcbeab5b37821e2b67a9f921749cd9a2e386d5a6d5bc6fc6d3 377d73dbb466e0bc7c9ee23166771b35ebdbe02ef17753d79fd3571d4ce659d7 @@ -158,16 +181,15 @@ ec1ec45792908e90484f7e629330666e7eee599f08729c93890a7205a6ba35f5 l ``` -The directory names do not correspond to the layer IDs (this has been true since -Docker 1.10). +The directory names do not correspond to the layer IDs. Now imagine that you have two different Dockerfiles. You use the first one to create an image called `acme/my-base-image:1.0`. ```dockerfile # syntax=docker/dockerfile:1 -FROM ubuntu:18.04 -COPY . /app +FROM alpine +RUN apk add --no-cache bash ``` The second one is based on `acme/my-base-image:1.0`, but has some additional @@ -176,16 +198,18 @@ layers: ```dockerfile # syntax=docker/dockerfile:1 FROM acme/my-base-image:1.0 +COPY . /app +RUN chmod +x /app/hello.sh CMD /app/hello.sh ``` -The second image contains all the layers from the first image, plus a new layer -with the `CMD` instruction, and a read-write container layer. Docker already -has all the layers from the first image, so it does not need to pull them again. -The two images share any layers they have in common. +The second image contains all the layers from the first image, plus new layers +created by the `COPY` and `RUN` instructions, and a read-write container layer. +Docker already has all the layers from the first image, so it does not need to +pull them again. The two images share any layers they have in common. If you build images from the two Dockerfiles, you can use `docker image ls` and -`docker history` commands to verify that the cryptographic IDs of the shared +`docker image history` commands to verify that the cryptographic IDs of the shared layers are the same. 1. Make a new directory `cow-test/` and change into it. @@ -193,16 +217,10 @@ layers are the same. 2. Within `cow-test/`, create a new file called `hello.sh` with the following contents: ```bash - #!/bin/sh + #!/usr/bin/env bash echo "Hello world" ``` - Save the file, and make it executable: - - ```bash - chmod +x hello.sh - ``` - 3. Copy the contents of the first Dockerfile above into a new file called `Dockerfile.base`. @@ -215,14 +233,23 @@ layers are the same. ```console $ docker build -t acme/my-base-image:1.0 -f Dockerfile.base . - Sending build context to Docker daemon 812.4MB - Step 1/2 : FROM ubuntu:18.04 - ---> d131e0fa2585 - Step 2/2 : COPY . /app - ---> Using cache - ---> bd09118bcef6 - Successfully built bd09118bcef6 - Successfully tagged acme/my-base-image:1.0 + [+] Building 6.0s (11/11) FINISHED + => [internal] load build definition from Dockerfile.base 0.4s + => => transferring dockerfile: 116B 0.0s + => [internal] load .dockerignore 0.3s + => => transferring context: 2B 0.0s + => resolve image config for docker.io/docker/dockerfile:1 1.5s + => [auth] docker/dockerfile:pull token for registry-1.docker.io 0.0s + => CACHED docker-image://docker.io/docker/dockerfile:1@sha256:9e2c9eca7367393aecc68795c671... 0.0s + => [internal] load .dockerignore 0.0s + => [internal] load build definition from Dockerfile.base 0.0s + => [internal] load metadata for docker.io/library/alpine:latest 0.0s + => CACHED [1/2] FROM docker.io/library/alpine 0.0s + => [2/2] RUN apk add --no-cache bash 3.1s + => exporting to image 0.2s + => => exporting layers 0.2s + => => writing image sha256:da3cf8df55ee9777ddcd5afc40fffc3ead816bda99430bad2257de4459625eaa 0.0s + => => naming to docker.io/acme/my-base-image:1.0 0.0s ``` 6. Build the second image. @@ -230,15 +257,25 @@ layers are the same. ```console $ docker build -t acme/my-final-image:1.0 -f Dockerfile . - Sending build context to Docker daemon 4.096kB - Step 1/2 : FROM acme/my-base-image:1.0 - ---> bd09118bcef6 - Step 2/2 : CMD /app/hello.sh - ---> Running in a07b694759ba - ---> dbf995fc07ff - Removing intermediate container a07b694759ba - Successfully built dbf995fc07ff - Successfully tagged acme/my-final-image:1.0 + [+] Building 3.6s (12/12) FINISHED + => [internal] load build definition from Dockerfile 0.1s + => => transferring dockerfile: 156B 0.0s + => [internal] load .dockerignore 0.1s + => => transferring context: 2B 0.0s + => resolve image config for docker.io/docker/dockerfile:1 0.5s + => CACHED docker-image://docker.io/docker/dockerfile:1@sha256:9e2c9eca7367393aecc68795c671... 0.0s + => [internal] load .dockerignore 0.0s + => [internal] load build definition from Dockerfile 0.0s + => [internal] load metadata for docker.io/acme/my-base-image:1.0 0.0s + => [internal] load build context 0.2s + => => transferring context: 340B 0.0s + => [1/3] FROM docker.io/acme/my-base-image:1.0 0.2s + => [2/3] COPY . /app 0.1s + => [3/3] RUN chmod +x /app/hello.sh 0.4s + => exporting to image 0.1s + => => exporting layers 0.1s + => => writing image sha256:8bd85c42fa7ff6b33902ada7dcefaaae112bf5673873a089d73583b0074313dd 0.0s + => => naming to docker.io/acme/my-final-image:1.0 0.0s ``` 7. Check out the sizes of the images: @@ -246,47 +283,99 @@ layers are the same. ```console $ docker image ls - REPOSITORY TAG IMAGE ID CREATED SIZE - acme/my-final-image 1.0 dbf995fc07ff 58 seconds ago 103MB - acme/my-base-image 1.0 bd09118bcef6 3 minutes ago 103MB + REPOSITORY TAG IMAGE ID CREATED SIZE + acme/my-final-image 1.0 8bd85c42fa7f About a minute ago 7.75MB + acme/my-base-image 1.0 da3cf8df55ee 2 minutes ago 7.75MB ``` -8. Check out the layers that comprise each image: +8. Check out the history of each image: ```console - $ docker history bd09118bcef6 - IMAGE CREATED CREATED BY SIZE COMMENT - bd09118bcef6 4 minutes ago /bin/sh -c #(nop) COPY dir:35a7eb158c1504e... 100B - d131e0fa2585 3 months ago /bin/sh -c #(nop) CMD ["/bin/bash"] 0B - 3 months ago /bin/sh -c mkdir -p /run/systemd && echo '... 7B - 3 months ago /bin/sh -c sed -i 's/^#\s*\(deb.*universe\... 2.78kB - 3 months ago /bin/sh -c rm -rf /var/lib/apt/lists/* 0B - 3 months ago /bin/sh -c set -xe && echo '#!/bin/sh' >... 745B - 3 months ago /bin/sh -c #(nop) ADD file:eef57983bd66e3a... 103MB + $ docker image history acme/my-base-image:1.0 + + IMAGE CREATED CREATED BY SIZE COMMENT + da3cf8df55ee 5 minutes ago RUN /bin/sh -c apk add --no-cache bash # bui… 2.15MB buildkit.dockerfile.v0 + 7 weeks ago /bin/sh -c #(nop) CMD ["/bin/sh"] 0B + 7 weeks ago /bin/sh -c #(nop) ADD file:f278386b0cef68136… 5.6MB ``` + Some steps do not have a size (`0B`), and are metadata-only changes, which do + not produce an image layer and do not take up any size, other than the metadata + itself. The output above shows that this image consists of 2 image layers. + ```console - $ docker history dbf995fc07ff + $ docker image history acme/my-final-image:1.0 - IMAGE CREATED CREATED BY SIZE COMMENT - dbf995fc07ff 3 minutes ago /bin/sh -c #(nop) CMD ["/bin/sh" "-c" "/a... 0B - bd09118bcef6 5 minutes ago /bin/sh -c #(nop) COPY dir:35a7eb158c1504e... 100B - d131e0fa2585 3 months ago /bin/sh -c #(nop) CMD ["/bin/bash"] 0B - 3 months ago /bin/sh -c mkdir -p /run/systemd && echo '... 7B - 3 months ago /bin/sh -c sed -i 's/^#\s*\(deb.*universe\... 2.78kB - 3 months ago /bin/sh -c rm -rf /var/lib/apt/lists/* 0B - 3 months ago /bin/sh -c set -xe && echo '#!/bin/sh' >... 745B - 3 months ago /bin/sh -c #(nop) ADD file:eef57983bd66e3a... 103MB + IMAGE CREATED CREATED BY SIZE COMMENT + 8bd85c42fa7f 3 minutes ago CMD ["/bin/sh" "-c" "/app/hello.sh"] 0B buildkit.dockerfile.v0 + 3 minutes ago RUN /bin/sh -c chmod +x /app/hello.sh # buil… 39B buildkit.dockerfile.v0 + 3 minutes ago COPY . /app # buildkit 222B buildkit.dockerfile.v0 + 4 minutes ago RUN /bin/sh -c apk add --no-cache bash # bui… 2.15MB buildkit.dockerfile.v0 + 7 weeks ago /bin/sh -c #(nop) CMD ["/bin/sh"] 0B + 7 weeks ago /bin/sh -c #(nop) ADD file:f278386b0cef68136… 5.6MB ``` - Notice that all the layers are identical except the top layer of the second - image. All the other layers are shared between the two images, and are only - stored once in `/var/lib/docker/`. The new layer actually doesn't take any - room at all, because it is not changing any files, but only running a command. + Notice that all steps of the first image are also included in the final + image. The final image includes the two layers from the first image, and + two layers that were added in the second image. - > **Note**: The `` lines in the `docker history` output indicate - > that those layers were built on another system and are not available - > locally. This can be ignored. + > What are the `` steps? + > + > The `` lines in the `docker history` output indicate that those + > steps were either built on another system and part of the `alpine` image + > that was pulled from Docker Hub, or were built with BuildKit as builder. + > Before BuildKit, the "classic" builder would produce a new "intermediate" + > image for each step for caching purposes, and the `IMAGE` column would show + > the ID of that image. + > BuildKit uses its own caching mechanism, and no longer requires intermediate + > images for caching. Refer to [build images with BuildKit](../../develop/develop-images/build_enhancements.md) + > to learn more about other enhancements made in BuildKit. + + +9. Check out the layers for each image + + Use the `docker image inspect` command to view the cryptographic IDs of the + layers in each image: + + {% raw %} + ```console + $ docker image inspect --format "{{json .RootFS.Layers}}" acme/my-final-image:1.0 + [ + "sha256:72e830a4dff5f0d5225cdc0a320e85ab1ce06ea5673acfe8d83a7645cbd0e9cf", + "sha256:07b4a9068b6af337e8b8f1f1dae3dd14185b2c0003a9a1f0a6fd2587495b204a" + ] + ``` + {% endraw %} + + {% raw %} + ```console + $ docker image inspect --format "{{json .RootFS.Layers}}" acme/my-base-image:1.0 + [ + "sha256:72e830a4dff5f0d5225cdc0a320e85ab1ce06ea5673acfe8d83a7645cbd0e9cf", + "sha256:07b4a9068b6af337e8b8f1f1dae3dd14185b2c0003a9a1f0a6fd2587495b204a", + "sha256:cc644054967e516db4689b5282ee98e4bc4b11ea2255c9630309f559ab96562e", + "sha256:e84fb818852626e89a09f5143dbc31fe7f0e0a6a24cd8d2eb68062b904337af4" + ] + ``` + {% endraw %} + + Notice that the first two layers are identical in both images. The second + image adds two additional layers. Shared image layers are only stored once + in `/var/lib/docker/` and are also shared when pushing and pulling and image + to an image registry. Shared image layers can therefore reduce network + bandwidth and storage. + + > Tip: format output of Docker commands with the `--format` option + > + > The examples above use the `docker image inspect` command with the `--format` + > option to view the layer IDs, formatted as a JSON array. The `--format` + > option on Docker commands can be a powerful feature that allows you to + > extract and format specific information from the output, without requiring + > additional tools such as `awk` or `sed`. To learn more about formatting + > the output of docker commands using the `--format` flag, refer to the + > [format command and log output section](../../config/formatting.md). + > We also pretty-printed the JSON output using the [`jq` utility](https://stedolan.github.io/jq/){: target="_blank" rel="noopener" class="_" } + > for readability. ### Copying makes containers efficient @@ -297,16 +386,14 @@ layer. This means that the writable layer is as small as possible. When an existing file in a container is modified, the storage driver performs a copy-on-write operation. The specifics steps involved depend on the specific -storage driver. For the `aufs`, `overlay`, and `overlay2` drivers, the +storage driver. For the `overlay2`, `overlay`, and `aufs` drivers, the copy-on-write operation follows this rough sequence: * Search through the image layers for the file to update. The process starts at the newest layer and works down to the base layer one layer at a time. When results are found, they are added to a cache to speed future operations. - * Perform a `copy_up` operation on the first copy of the file that is found, to copy the file to the container's writable layer. - * Any modifications are made to this copy of the file, and the container cannot see the read-only copy of the file that exists in the lower layer. @@ -316,13 +403,20 @@ descriptions. Containers that write a lot of data consume more space than containers that do not. This is because most write operations consume new space in the -container's thin writable top layer. +container's thin writable top layer. Note that changing the metadata of files, +for example, changing file permissions or ownership of a file, can also result +in a `copy_up` operation, therefore duplicating the file to the writable layer. -> **Note**: for write-heavy applications, you should not store the data in -> the container. Instead, use Docker volumes, which are independent of the -> running container and are designed to be efficient for I/O. In addition, -> volumes can be shared among containers and do not increase the size of your -> container's writable layer. +> Tip: Use volumes for write-heavy applications +> +> For write-heavy applications, you should not store the data in the container. +> Applications, such as write-intensive database storage, are known to be +> problematic particularly when pre-existing data exists in the read-only layer. +> +> Instead, use Docker volumes, which are independent of the running container, +> and designed to be efficient for I/O. In addition, volumes can be shared among +> containers and do not increase the size of your container's writable layer. +> Refer to the [use volumes](../volumes.md) section to learn about volumes. A `copy_up` operation can incur a noticeable performance overhead. This overhead is different depending on which storage driver is in use. Large files, @@ -334,72 +428,107 @@ To verify the way that copy-on-write works, the following procedures spins up 5 containers based on the `acme/my-final-image:1.0` image we built earlier and examines how much room they take up. -> **Note**: This procedure doesn't work on Docker Desktop for Mac or Docker Desktop for Windows. - 1. From a terminal on your Docker host, run the following `docker run` commands. The strings at the end are the IDs of each container. - ```bash + ```console $ docker run -dit --name my_container_1 acme/my-final-image:1.0 bash \ && docker run -dit --name my_container_2 acme/my-final-image:1.0 bash \ && docker run -dit --name my_container_3 acme/my-final-image:1.0 bash \ && docker run -dit --name my_container_4 acme/my-final-image:1.0 bash \ && docker run -dit --name my_container_5 acme/my-final-image:1.0 bash - c36785c423ec7e0422b2af7364a7ba4da6146cbba7981a0951fcc3fa0430c409 - dcad7101795e4206e637d9358a818e5c32e13b349e62b00bf05cd5a4343ea513 - 1e7264576d78a3134fbaf7829bc24b1d96017cf2bc046b7cd8b08b5775c33d0c - 38fa94212a419a082e6a6b87a8e2ec4a44dd327d7069b85892a707e3fc818544 - 1a174fc216cccf18ec7d4fe14e008e30130b11ede0f0f94a87982e310cf2e765 + 40ebdd7634162eb42bdb1ba76a395095527e9c0aa40348e6c325bd0aa289423c + a5ff32e2b551168b9498870faf16c9cd0af820edf8a5c157f7b80da59d01a107 + 3ed3c1a10430e09f253704116965b01ca920202d52f3bf381fbb833b8ae356bc + 939b3bf9e7ece24bcffec57d974c939da2bdcc6a5077b5459c897c1e2fa37a39 + cddae31c314fbab3f7eabeb9b26733838187abc9a2ed53f97bd5b04cd7984a5a ``` +2. Run the `docker ps` command with the `--size` option to verify the 5 containers + are running, and to see each container's size. -2. Run the `docker ps` command to verify the 5 containers are running. + {% raw %} + ```console + $ docker ps --size --format "table {{.ID}}\t{{.Image}}\t{{.Names}}\t{{.Size}}" - ```bash - CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES - 1a174fc216cc acme/my-final-image:1.0 "bash" About a minute ago Up About a minute my_container_5 - 38fa94212a41 acme/my-final-image:1.0 "bash" About a minute ago Up About a minute my_container_4 - 1e7264576d78 acme/my-final-image:1.0 "bash" About a minute ago Up About a minute my_container_3 - dcad7101795e acme/my-final-image:1.0 "bash" About a minute ago Up About a minute my_container_2 - c36785c423ec acme/my-final-image:1.0 "bash" About a minute ago Up About a minute my_container_1 + CONTAINER ID IMAGE NAMES SIZE + cddae31c314f acme/my-final-image:1.0 my_container_5 0B (virtual 7.75MB) + 939b3bf9e7ec acme/my-final-image:1.0 my_container_4 0B (virtual 7.75MB) + 3ed3c1a10430 acme/my-final-image:1.0 my_container_3 0B (virtual 7.75MB) + a5ff32e2b551 acme/my-final-image:1.0 my_container_2 0B (virtual 7.75MB) + 40ebdd763416 acme/my-final-image:1.0 my_container_1 0B (virtual 7.75MB) ``` + {% endraw %} + The output above shows that all containers share the image's read-only layers + (7.75MB), but no data was written to the container's filesystem, so no additional + storage is used for the containers. -3. List the contents of the local storage area. + > Advanced: metadata and logs storage used for containers + > + > **Note**: This step requires a Linux machine, and does not work on Docker + > Desktop for Mac or Docker Desktop for Windows, as it requires access to + > the Docker Daemon's file storage. + > + > While the output of `docker ps` provides you information about disk space + > consumed by a container's writable layer, it does not include information + > about metadata and log-files stored for each container. + > + > More details can be obtained my exploring the Docker Daemon's storage location + > (`/var/lib/docker` by default). + > + > ```console + > $ sudo du -sh /var/lib/docker/containers/* + > + > 36K /var/lib/docker/containers/3ed3c1a10430e09f253704116965b01ca920202d52f3bf381fbb833b8ae356bc + > 36K /var/lib/docker/containers/40ebdd7634162eb42bdb1ba76a395095527e9c0aa40348e6c325bd0aa289423c + > 36K /var/lib/docker/containers/939b3bf9e7ece24bcffec57d974c939da2bdcc6a5077b5459c897c1e2fa37a39 + > 36K /var/lib/docker/containers/a5ff32e2b551168b9498870faf16c9cd0af820edf8a5c157f7b80da59d01a107 + > 36K /var/lib/docker/containers/cddae31c314fbab3f7eabeb9b26733838187abc9a2ed53f97bd5b04cd7984a5a + > ``` + > + > Each of these containers only takes up 36k of space on the filesystem. - ```bash - $ sudo ls /var/lib/docker/containers +3. Per-container storage - 1a174fc216cccf18ec7d4fe14e008e30130b11ede0f0f94a87982e310cf2e765 - 1e7264576d78a3134fbaf7829bc24b1d96017cf2bc046b7cd8b08b5775c33d0c - 38fa94212a419a082e6a6b87a8e2ec4a44dd327d7069b85892a707e3fc818544 - c36785c423ec7e0422b2af7364a7ba4da6146cbba7981a0951fcc3fa0430c409 - dcad7101795e4206e637d9358a818e5c32e13b349e62b00bf05cd5a4343ea513 + To demonstrate this, run the following command to write the word 'hello' to + a file on the container's writable layer in containers `my_container_1`, + `my_container_2`, and `my_container_3`: + + ```console + $ for i in {1..3}; do docker exec my_container_$i sh -c 'printf hello > /out.txt'; done ``` + + Running the `docker ps` command again afterward shows that those containers + now consume 5 bytes each. This data is unique to each container, and not + shared. The read-only layers of the containers are not affected, and are still + shared by all containers. -4. Now check out their sizes: + {% raw %} + ```console + $ docker ps --size --format "table {{.ID}}\t{{.Image}}\t{{.Names}}\t{{.Size}}" - ```bash - $ sudo du -sh /var/lib/docker/containers/* - - 32K /var/lib/docker/containers/1a174fc216cccf18ec7d4fe14e008e30130b11ede0f0f94a87982e310cf2e765 - 32K /var/lib/docker/containers/1e7264576d78a3134fbaf7829bc24b1d96017cf2bc046b7cd8b08b5775c33d0c - 32K /var/lib/docker/containers/38fa94212a419a082e6a6b87a8e2ec4a44dd327d7069b85892a707e3fc818544 - 32K /var/lib/docker/containers/c36785c423ec7e0422b2af7364a7ba4da6146cbba7981a0951fcc3fa0430c409 - 32K /var/lib/docker/containers/dcad7101795e4206e637d9358a818e5c32e13b349e62b00bf05cd5a4343ea513 + CONTAINER ID IMAGE NAMES SIZE + cddae31c314f acme/my-final-image:1.0 my_container_5 0B (virtual 7.75MB) + 939b3bf9e7ec acme/my-final-image:1.0 my_container_4 0B (virtual 7.75MB) + 3ed3c1a10430 acme/my-final-image:1.0 my_container_3 5B (virtual 7.75MB) + a5ff32e2b551 acme/my-final-image:1.0 my_container_2 5B (virtual 7.75MB) + 40ebdd763416 acme/my-final-image:1.0 my_container_1 5B (virtual 7.75MB) ``` + {% endraw %} - Each of these containers only takes up 32k of space on the filesystem. - -Not only does copy-on-write save space, but it also reduces start-up time. -When you start a container (or multiple containers from the same image), Docker -only needs to create the thin writable container layer. +The examples above illustrate how copy-on-write filesystems help making containers +efficient. Not only does copy-on-write save space, but it also reduces container +start-up time. When you create a container (or multiple containers from the same +image), Docker only needs to create the thin writable container layer. If Docker had to make an entire copy of the underlying image stack each time it -started a new container, container start times and disk space used would be +created a new container, container create times and disk space used would be significantly increased. This would be similar to the way that virtual machines -work, with one or more virtual disks per virtual machine. +work, with one or more virtual disks per virtual machine. The [`vfs` storage](vfs-driver.md) +does not provide a CoW filesystem or other optimizations. When using this storage +driver, a full copy of the image's data is created for each container. ## Related information diff --git a/storage/storagedriver/select-storage-driver.md b/storage/storagedriver/select-storage-driver.md index 8b2dc3618a..171037c57a 100644 --- a/storage/storagedriver/select-storage-driver.md +++ b/storage/storagedriver/select-storage-driver.md @@ -13,90 +13,63 @@ use Docker volumes to write data. However, some workloads require you to be able to write to the container's writable layer. This is where storage drivers come in. -Docker supports several different storage drivers, using a pluggable -architecture. The storage driver controls how images and containers are stored -and managed on your Docker host. - -After you have read the [storage driver overview](index.md), the -next step is to choose the best storage driver for your workloads. In making -this decision, there are three high-level factors to consider: - -If multiple storage drivers are supported in your kernel, Docker has a prioritized -list of which storage driver to use if no storage driver is explicitly configured, -assuming that the storage driver meets the prerequisites. - -Use the storage driver with the best overall performance and stability in the most -usual scenarios. +Docker supports several storage drivers, using a pluggable architecture. The +storage driver controls how images and containers are stored and managed on your +Docker host. After you have read the [storage driver overview](index.md), the +next step is to choose the best storage driver for your workloads. Use the storage +driver with the best overall performance and stability in the most usual scenarios. -Docker supports the following storage drivers: +The Docker Engine provides the following storage drivers on Linux: -* `overlay2` is the preferred storage driver, for all currently supported - Linux distributions, and requires no extra configuration. -* `aufs` was the preferred storage driver for Docker 18.06 and older, when - running on Ubuntu 14.04 on kernel 3.13 which had no support for `overlay2`. -* `fuse-overlayfs` is preferred only for running Rootless Docker - on a host that does not provide support for rootless `overlay2`. - On Ubuntu and Debian 10, the `fuse-overlayfs` driver does not need to be - used `overlay2` works even in rootless mode. - See [Rootless mode documentation](../../engine/security/rootless.md). -* `devicemapper` is supported, but requires `direct-lvm` for production - environments, because `loopback-lvm`, while zero-configuration, has very - poor performance. `devicemapper` was the recommended storage driver for - CentOS and RHEL, as their kernel version did not support `overlay2`. However, - current versions of CentOS and RHEL now have support for `overlay2`, - which is now the recommended driver. - * The `btrfs` and `zfs` storage drivers are used if they are the backing - filesystem (the filesystem of the host on which Docker is installed). - These filesystems allow for advanced options, such as creating "snapshots", - but require more maintenance and setup. Each of these relies on the backing - filesystem being configured correctly. - * The `vfs` storage driver is intended for testing purposes, and for situations - where no copy-on-write filesystem can be used. Performance of this storage - driver is poor, and is not generally recommended for production use. +| Driver | Description | +|--------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `overlay2` | `overlay2` is the preferred storage driver for all currently supported Linux distributions, and requires no extra configuration. | +| `fuse-overlayfs` | `fuse-overlayfs`is preferred only for running Rootless Docker on a host that does not provide support for rootless `overlay2`. On Ubuntu and Debian 10, the `fuse-overlayfs` driver does not need to be used, and `overlay2` works even in rootless mode. Refer to the [rootless mode documentation](../../engine/security/rootless.md) for details. | +| `btrfs` and `zfs` | The `btrfs` and `zfs` storage drivers allow for advanced options, such as creating "snapshots", but require more maintenance and setup. Each of these relies on the backing filesystem being configured correctly. | +| `vfs` | The `vfs` storage driver is intended for testing purposes, and for situations where no copy-on-write filesystem can be used. Performance of this storage driver is poor, and is not generally recommended for production use. | +| `aufs` | The `aufs` storage driver Was the preferred storage driver for Docker 18.06 and older, when running on Ubuntu 14.04 on kernel 3.13 which had no support for `overlay2`. However, current versions of Ubuntu and Debian now have support for `overlay2`, which is now the recommended driver. | +| `devicemapper` | The `devicemapper` storage driver requires `direct-lvm` for production environments, because `loopback-lvm`, while zero-configuration, has very poor performance. `devicemapper` was the recommended storage driver for CentOS and RHEL, as their kernel version did not support `overlay2`. However, current versions of CentOS and RHEL now have support for `overlay2`, which is now the recommended driver. | +| `overlay` | The legacy `overlay` driver was used for kernels that did not support the "multiple-lowerdir" feature required for `overlay2` All currently supported Linux distributions now provide support for this, and it is therefore deprecated. | -Docker's source code defines the selection order. You can see the order at -[the source code for Docker Engine {{ site.docker_ce_version }}](https://github.com/moby/moby/blob/{{ site.docker_ce_version }}/daemon/graphdriver/driver_linux.go#L52-L53) - -If you run a different version of Docker, you can use the branch selector at the top of the file viewer to choose a different branch. +The Docker Engine has a prioritized list of which storage driver to use if no +storage driver is explicitly configured, assuming that the storage driver meets +the prerequisites, and automatically selects a compatible storage driver. You +can see the order in the [source code for Docker Engine {{ site.docker_ce_version }}](https://github.com/moby/moby/blob/{{ site.docker_ce_version }}/daemon/graphdriver/driver_linux.go#L52-L53). {: id="storage-driver-order" } -Some storage drivers require you to use a specific format for the backing filesystem. If you have external -requirements to use a specific backing filesystem, this may limit your choices. See [Supported backing filesystems](#supported-backing-filesystems). +Some storage drivers require you to use a specific format for the backing filesystem. +If you have external requirements to use a specific backing filesystem, this may +limit your choices. See [Supported backing filesystems](#supported-backing-filesystems). -After you have narrowed down which storage drivers you can choose from, your choice is determined by the -characteristics of your workload and the level of stability you need. See [Other considerations](#other-considerations) -for help in making the final decision. - -> ***NOTE***: Your choice may be limited by your operating system and distribution. -> For instance, `aufs` is only supported on Ubuntu and Debian, and may require extra packages -> to be installed, while `btrfs` is only supported on SLES, which is only supported with Docker -> Enterprise. See [Support storage drivers per Linux distribution](#supported-storage-drivers-per-linux-distribution) -> for more information. +After you have narrowed down which storage drivers you can choose from, your choice +is determined by the characteristics of your workload and the level of stability +you need. See [Other considerations](#other-considerations) for help in making +the final decision. ## Supported storage drivers per Linux distribution -At a high level, the storage drivers you can use is partially determined by -the Docker edition you use. +> Docker Desktop, and Docker in Rootless mode +> +> Modifying the storage-driver is not supported on Docker Desktop for Mac and +> Docker Desktop for Windows, and only the default storage driver can be used. +> The comparison table below is also not applicable for Rootless mode. For the +> drivers available in rootless mode, see the [Rootless mode documentation](../../engine/security/rootless.md). -In addition, Docker does not recommend any configuration that requires you to -disable security features of your operating system, such as the need to disable -`selinux` if you use the `overlay` or `overlay2` driver on CentOS. +Your operating system and kernel may not support every storage driver. For +instance, `aufs` is only supported on Ubuntu and Debian, and may require extra +packages to be installed, while `btrfs` is only supported if your system uses +`btrfs` as storage. In general, the following configurations work on recent +versions of the Linux distribution: -### Docker Engine - Community - -For Docker Engine - Community, only some configurations are tested, and your operating -system's kernel may not support every storage driver. In general, the following -configurations work on recent versions of the Linux distribution: - -| Linux distribution | Recommended storage drivers | Alternative drivers | -|:--------------------|:-----------------------------------------------------------------------|:--------------------------------------------------| -| Docker Engine - Community on Ubuntu | `overlay2` or `aufs` (for Ubuntu 14.04 running on kernel 3.13) | `overlay`¹, `devicemapper`², `zfs`, `vfs` | -| Docker Engine - Community on Debian | `overlay2` (Debian Stretch), `aufs` or `devicemapper` (older versions) | `overlay`¹, `vfs` | -| Docker Engine - Community on CentOS | `overlay2` | `overlay`¹, `devicemapper`², `zfs`, `vfs` | -| Docker Engine - Community on Fedora | `overlay2` | `overlay`¹, `devicemapper`², `zfs`, `vfs` | -| Docker Engine - Community on SLES 15 | `overlay2` | `overlay`¹, `devicemapper`², `vfs` | -| Docker Engine - Community on RHEL | `overlay2` | `overlay`¹, `devicemapper`², `vfs` | +| Linux distribution | Recommended storage drivers | Alternative drivers | +|:--------------------|:------------------------------|:---------------------------------------------------| +| Ubuntu | `overlay2` | `overlay`¹, `devicemapper`², `aufs`³, `zfs`, `vfs` | +| Debian | `overlay2` | `overlay`¹, `devicemapper`², `aufs`³, `vfs` | +| CentOS | `overlay2` | `overlay`¹, `devicemapper`², `zfs`, `vfs` | +| Fedora | `overlay2` | `overlay`¹, `devicemapper`², `zfs`, `vfs` | +| SLES 15 | `overlay2` | `overlay`¹, `devicemapper`², `vfs` | +| RHEL | `overlay2` | `overlay`¹, `devicemapper`², `vfs` | ¹) The `overlay` storage driver is deprecated, and will be removed in a future release. It is recommended that users of the `overlay` storage driver migrate to `overlay2`. @@ -105,49 +78,33 @@ release. It is recommended that users of the `overlay` storage driver migrate to release. It is recommended that users of the `devicemapper` storage driver migrate to `overlay2`. -> **Note** -> -> The comparison table above is not applicable for Rootless mode. -> For the drivers available in Rootless mode, see [the Rootless mode documentation](../../engine/security/rootless.md). - -When possible, `overlay2` is the recommended storage driver. When installing -Docker for the first time, `overlay2` is used by default. Previously, `aufs` was -used by default when available, but this is no longer the case. If you want to -use `aufs` on new installations going forward, you need to explicitly configure -it, and you may need to install extra packages, such as `linux-image-extra`. -See [aufs](aufs-driver.md). - -On existing installations using `aufs`, it is still used. +³) The `aufs` storage driver is deprecated, and will be removed in a future +release. It is recommended that users of the `aufs` storage driver migrate +to `overlay2`. When in doubt, the best all-around configuration is to use a modern Linux distribution with a kernel that supports the `overlay2` storage driver, and to use Docker volumes for write-heavy workloads instead of relying on writing data to the container's writable layer. -The `vfs` storage driver is usually not the best choice. Before using the `vfs` -storage driver, be sure to read about +The `vfs` storage driver is usually not the best choice, and primarily intended +for debugging purposes in situations where no other storage-driver is supported. +Before using the `vfs` storage driver, be sure to read about [its performance and storage characteristics and limitations](vfs-driver.md). -> **Expectations for non-recommended storage drivers**: Commercial support is -> not available for Docker Engine - Community, and you can technically use any storage driver -> that is available for your platform. For instance, you can use `btrfs` with -> Docker Engine - Community, even though it is not recommended on any platform for -> Docker Engine - Community, and you do so at your own risk. -> -> The recommendations in the table above are based on automated regression -> testing and the configurations that are known to work for a large number of -> users. If you use a recommended configuration and find a reproducible issue, -> it is likely to be fixed very quickly. If the driver that you want to use is -> not recommended according to this table, you can run it at your own risk. You -> can and should still report any issues you run into. However, such issues -> have a lower priority than issues encountered when using a recommended -> configuration. +The recommendations in the table above are known to work for a large number of +users. If you use a recommended configuration and find a reproducible issue, +it is likely to be fixed very quickly. If the driver that you want to use is +not recommended according to this table, you can run it at your own risk. You +can and should still report any issues you run into. However, such issues +have a lower priority than issues encountered when using a recommended +configuration. -### Docker Desktop for Mac and Docker Desktop for Windows - -Docker Desktop for Mac and Docker Desktop for Windows are intended for development, rather -than production. Modifying the storage driver on these platforms is not -possible. +Depending on your Linux distribution, other storage-drivers, such as `btrfs` may +be available. These storage drivers can have advantages for specific use-cases, +but may require additional set-up or maintenance, which make them not recommended +for common scenarios. Refer to the documentation for those storage drivers for +details. ## Supported backing filesystems @@ -205,8 +162,8 @@ specific shared storage system. For some users, stability is more important than performance. Though Docker considers all of the storage drivers mentioned here to be stable, some are newer -and are still under active development. In general, `overlay2`, `aufs`, `overlay`, -and `devicemapper` are the choices with the highest stability. +and are still under active development. In general, `overlay2`, `aufs`, and +`devicemapper` are the choices with the highest stability. ### Test with your own workloads @@ -223,7 +180,7 @@ set-up steps to use a given storage driver. To see what storage driver Docker is currently using, use `docker info` and look for the `Storage Driver` line: -```bash +```console $ docker info Containers: 0 @@ -237,11 +194,13 @@ To change the storage driver, see the specific instructions for the new storage driver. Some drivers require additional configuration, including configuration to physical or logical disks on the Docker host. -> **Important**: When you change the storage driver, any existing images and -> containers become inaccessible. This is because their layers cannot be used -> by the new storage driver. If you revert your changes, you can -> access the old images and containers again, but any that you pulled or -> created using the new driver are then inaccessible. +> **Important** +> +> When you change the storage driver, any existing images and containers become +> inaccessible. This is because their layers cannot be used by the new storage +> driver. If you revert your changes, you can access the old images and containers +> again, but any that you pulled or created using the new driver are then +> inaccessible. {:.important} ## Related information