From 54e13c08d90efecbc62243bd4b99b0e9c937c752 Mon Sep 17 00:00:00 2001 From: David Karlsson <35727626+dvdksn@users.noreply.github.com> Date: Tue, 10 Oct 2023 09:28:55 +0200 Subject: [PATCH] build: refresh building best practices Signed-off-by: David Karlsson <35727626+dvdksn@users.noreply.github.com> --- content/build/building/context.md | 465 +++++++++++++------ content/build/building/packaging.md | 6 +- content/build/cache/_index.md | 2 +- content/build/hydrobuild.md | 6 +- content/develop/develop-images/guidelines.md | 268 ++--------- content/get-started/09_image_best.md | 4 +- content/network/proxy.md | 2 +- data/toc.yaml | 2 +- 8 files changed, 369 insertions(+), 386 deletions(-) diff --git a/content/build/building/context.md b/content/build/building/context.md index a710e3c3f3..f43bfccb22 100644 --- a/content/build/building/context.md +++ b/content/build/building/context.md @@ -4,48 +4,57 @@ description: Learn how to use the build context to access files from your Docker keywords: build, buildx, buildkit, context, git, tarball, stdin --- -The [`docker build`](../../engine/reference/commandline/build.md) or -[`docker buildx build`](../../engine/reference/commandline/buildx_build.md) -commands build Docker images from a [Dockerfile](../../engine/reference/builder.md) -and a "context". +The `docker build` and `docker buildx build` commands build Docker images from +a [Dockerfile](../../engine/reference/builder.md) and a context. -The build context is the argument that you pass to the build command: +## What is a build context? + +The build context is the set of files that your build can access. +The positional argument that you pass to the build command specifies the +context that you want to use for the build: ```console $ docker build [OPTIONS] PATH | URL | - ^^^^^^^^^^^^^^ ``` -## What is a build context? - You can pass any of the following inputs as the context for a build: - The relative or absolute path to a local directory -- The address of a remote Git repository, tarball, or plain-text file -- A piped plain-text file or a tarball using standard input +- A remote URL of a Git repository, tarball, or plain-text file +- A plain-text file or tarball piped to the `docker build` command through standard input ### Filesystem contexts -When your build context is a local directory, a remote Git repository, or a tar file, -then that becomes the set of files that the builder can access during the build. -Build instructions can refer to any of the files and directories in the context. -For example, when you use a [`COPY` instruction](../../engine/reference/builder.md#copy), -the builder copies the file or directory from the build context, into the build container. +When your build context is a local directory, a remote Git repository, or a tar +file, then that becomes the set of files that the builder can access during the +build. Build instructions such as `COPY` and `ADD` can refer to any of the +files and directories in the context. + A filesystem build context is processed recursively: - When you specify a local directory or a tarball, all subdirectories are included - When you specify a remote Git repository, the repository and all submodules are included +For more information about the different types of filesystem contexts that you +can use with your builds, see: + +- [Local files](#local-context) +- [Git repositories](#git-repositories) +- [Remote tarballs](#remote-tarballs) + ### Text file contexts When your build context is a plain-text file, the builder interprets the file -as a Dockerfile. With this approach, the builder doesn't receive a filesystem context. -For more information about building with a text file context, see [Text files](#text-files). +as a Dockerfile. With this approach, the build doesn't use a filesystem context. -## Local directories and tarballs +For more information, see [empty build context](#empty-context). -The following example shows a build command that uses the current directory -(`.`) as a build context: +## Local context + +To use a local build context, you can specify a relative or absolute filepath +to the `docker build` command. The following example shows a build command that +uses the current directory (`.`) as a build context: ```console $ docker build . @@ -56,11 +65,16 @@ $ docker build . ... ``` -This makes files and directories in the current working directory available -to the builder. The builder loads the files that it needs from the build context, -when it needs them. +This makes files and directories in the current working directory available to +the builder. The builder loads the files it needs from the build context when +needed. -For example, given the following directory structure: +You can also use local tarballs as build context, by piping the tarball +contents to the `docker build` command. See [Tarballs](#local-tarballs). + +### Local directories + +Consider the following directory structure: ``` . @@ -71,8 +85,8 @@ For example, given the following directory structure: └── package-lock.json ``` -Dockerfile instructions can reference and include these files in the build -if you pass the directory as a context. +Dockerfile instructions can reference and include these files in the build if +you pass this directory as a context. ```dockerfile # syntax=docker/dockerfile:1 @@ -83,46 +97,93 @@ RUN npm ci COPY index.ts src . ``` -### `.dockerignore` - -You can use a [`.dockerignore`](../../engine/reference/builder.md#dockerignore-file) -file to exclude some files or directories from being sent: - -```gitignore -# .dockerignore -node_modules -bar +```console +$ docker build . ``` -A `.dockerignore` file located at the root of build context is automatically -detected and used. +### Local context with Dockerfile from stdin -If you use multiple Dockerfiles, you can use different ignore-files for each -Dockerfile. You do so using a special naming convention for the ignore-files. -Place your ignore-file in the same directory as the Dockerfile, and prefix the -ignore-file with the name of the Dockerfile. - -For example: +Use the following syntax to build an image using files on your local +filesystem, while using a Dockerfile from stdin. ```console -. -├── index.ts -├── src/ -├── docker -│   ├── build.Dockerfile -│   ├── build.Dockerfile.dockerignore -│   ├── lint.Dockerfile -│   ├── lint.Dockerfile.dockerignore -│   ├── test.Dockerfile -│   └── test.Dockerfile.dockerignore -├── package.json -└── package-lock.json +$ docker build -f- ``` -A Dockerfile-specific ignore-file takes precedence over the `.dockerignore` -file at the root of the build context if both exist. +The syntax uses the -f (or --file) option to specify the Dockerfile to use, and +it uses a hyphen (-) as filename to instruct Docker to read the Dockerfile from +stdin. -## Git repositories +The following example uses the current directory (.) as the build context, and +builds an image using a Dockerfile passed through stdin using a here-document. + +```bash +# create a directory to work in +mkdir example +cd example + +# create an example file +touch somefile.txt + +# build an image using the current directory as context +# and a Dockerfile passed through stdin +docker build -t myimage:latest -f- . < docker buildx build \ > **Note** > -> Don't use `--build-arg` for secrets, except for -> [HTTP proxies](../../network/proxy.md#set-proxy-using-the-cli) +> Don't use `--build-arg` for secrets. + +### Remote context with Dockerfile from stdin + +Use the following syntax to build an image using files on your local +filesystem, while using a Dockerfile from stdin. + +```console +$ docker build -f- +``` + +The syntax uses the -f (or --file) option to specify the Dockerfile to use, and +it uses a hyphen (-) as filename to instruct Docker to read the Dockerfile from +stdin. + +This can be useful in situations where you want to build an image from a +repository that doesn't contain a Dockerfile. Or if you want to build with a +custom Dockerfile, without maintaining your own fork of the repository. + +The following example builds an image using a Dockerfile from stdin, and adds +the `hello.c` file from the [hello-world](https://github.com/docker-library/hello-world) +repository on GitHub. + +```bash +docker build -t myimage:latest -f- https://github.com/docker-library/hello-world.git <}} +{{< tab name="Unix pipe" >}} + ```console $ docker build - < Dockerfile ``` -With PowerShell on Windows, you can run: +{{< /tab >}} +{{< tab name="PowerShell" >}} ```powershell Get-Content Dockerfile | docker build - ``` -To use a remote text file, pass the URL of the text file as the argument to the -build command: +{{< /tab >}} +{{< tab name="Heredocs" >}} + +```bash +docker build -t myimage:latest - <}} +{{< tab name="Remote file" >}} ```console $ docker build https://raw.githubusercontent.com/dvdksn/clockbox/main/Dockerfile ``` -Again, this means that the build has no filesystem context, -so Dockerfile commands such as `COPY` can't refer to local files: +{{< /tab >}} +{{< /tabs >}} + +When you build without a filesystem context, Dockerfile instructions such as +`COPY` can't refer to local files: ```console $ ls @@ -352,18 +406,143 @@ Dockerfile:2 ERROR: failed to solve: failed to compute cache key: failed to calculate checksum of ref 7ab2bb61-0c28-432e-abf5-a4c3440bc6b6::4lgfpdf54n5uqxnv9v6ymg7ih: "/main.c": not found ``` -#### Build using heredocs +## .dockerignore files -The following example builds an image using a `Dockerfile` that is passed -through standard input using -[shell heredocs](https://en.wikipedia.org/wiki/Here_document): +You can use a `.dockerignore` file to exclude files or directories from the +build context. -```bash -docker build -t myimage:latest - < **Note** +> +> For historical reasons, the pattern `.` is ignored. + +Beyond Go's `filepath.Match` rules, Docker also supports a special wildcard +string `**` that matches any number of directories (including zero). For +example, `**/*.go` excludes all files that end with `.go` found anywhere in the +build context. + +You can use the `.dockerignore` file to exclude the `Dockerfile` and +`.dockerignore` files. These files are still sent to the builder as they're +needed for running the build. But you can't copy the files into the image using +`ADD`, `COPY`, or bind mounts. + +#### Negating matches + +You can prepend lines with a `!` (exclamation mark) to make exceptions to +exclusions. The following is an example `.dockerignore` file that uses this +mechanism: + +```gitignore +*.md +!README.md +``` + +All markdown files right under the context directory _except_ `README.md` are +excluded from the context. Note that markdown files under subdirectories are +still included. + +The placement of `!` exception rules influences the behavior: the last line of +the `.dockerignore` that matches a particular file determines whether it's +included or excluded. Consider the following example: + +```gitignore +*.md +!README*.md +README-secret.md +``` + +No markdown files are included in the context except README files other than +`README-secret.md`. + +Now consider this example: + +```gitignore +*.md +README-secret.md +!README*.md +``` + +All of the README files are included. The middle line has no effect because +`!README*.md` matches `README-secret.md` and comes last. diff --git a/content/build/building/packaging.md b/content/build/building/packaging.md index 9ce8c79c87..2f230637c2 100644 --- a/content/build/building/packaging.md +++ b/content/build/building/packaging.md @@ -198,7 +198,7 @@ The next instruction uses the COPY hello.py / ``` -A [build context](context.md) is the set of files that you can access +A [build context](./context.md) is the set of files that you can access in Dockerfile instructions such as `COPY` and `ADD`. After the `COPY` instruction, the `hello.py` file is added to the filesystem @@ -253,7 +253,7 @@ $ docker build -t test:latest . The `-t test:latest` option specifies the name and tag of the image. The single dot (`.`) at the end of the command sets the -[build context](context.md) to the current directory. This means that the +[build context](./context.md) to the current directory. This means that the build expects to find the Dockerfile and the `hello.py` file in the directory where the command is invoked. If those files aren't there, the build fails. @@ -273,4 +273,4 @@ If you are interested in examples in other languages, such as Go, check out our [language-specific guides](../../language/index.md) in the Guides section. For more information about building, including advanced use cases and patterns, -refer to the [Build with Docker](../guide/index.md) guide. \ No newline at end of file +refer to the [Build with Docker](../guide/index.md) guide. diff --git a/content/build/cache/_index.md b/content/build/cache/_index.md index 857e063de9..42507216fd 100644 --- a/content/build/cache/_index.md +++ b/content/build/cache/_index.md @@ -144,7 +144,7 @@ COPY . /src ``` You can also create a -[`.dockerignore` file](../../engine/reference/builder.md#dockerignore-file), +[`.dockerignore` file](../../build/building/context.md#dockerignore-files), and use that to specify which files and directories to exclude from the build context. diff --git a/content/build/hydrobuild.md b/content/build/hydrobuild.md index 4d70d69b31..00fbe886d8 100644 --- a/content/build/hydrobuild.md +++ b/content/build/hydrobuild.md @@ -391,10 +391,10 @@ more efficient: ### Dockerignore files -Using a [`.dockerignore` file](./building/context.md#dockerignore), you can be +Using a [`.dockerignore` file](./building/context.md#dockerignore-files), you can be explicit about which local files that you don’t want to include in the build -context. Files caught by the [glob patterns](../engine/reference/builder.md#dockerignore-file) -you specify in your ignore-file are not transferred to the remote builder. +context. Files caught by the glob patterns you specify in your ignore-file are +not transferred to the remote builder. Some examples of things you might want to add to your `.dockerignore` file are: diff --git a/content/develop/develop-images/guidelines.md b/content/develop/develop-images/guidelines.md index 5b83e09ae3..294665a96c 100644 --- a/content/develop/develop-images/guidelines.md +++ b/content/develop/develop-images/guidelines.md @@ -1,9 +1,29 @@ --- description: Hints, tips and guidelines for writing clean, reliable Dockerfiles keywords: parent image, images, dockerfile, best practices, hub, official image -title: General best practices for writing Dockerfiles +title: General best practices for writing Dockerfiles --- +## Use multi-stage builds + +Multi-stage builds let you reduce the size of your final image, by creating a +cleaner separation between the building of your image and the final output. +Split your Dockerfile instructions into distinct stages to make sure that the +resulting output only contains the files that's needed to run the application. + +Using multiple stages can also let you build more efficiently by executing +build steps in parallel. + +See [Multi-stage builds](../../build/building/multi-stage.md) for more +information. + +## Exclude with .dockerignore + +To exclude files not relevant to the build, without restructuring your source +repository, use a `.dockerignore` file. This file supports exclusion patterns +similar to `.gitignore` files. For information on creating one, see +[Dockerignore file](../../build/building/context.md#dockerignore-files). + ## Create ephemeral containers The image defined by your Dockerfile should generate containers that are as @@ -15,13 +35,6 @@ Refer to [Processes](https://12factor.net/processes) under _The Twelve-factor Ap methodology to get a feel for the motivations of running containers in such a stateless fashion. -## Exclude with .dockerignore - -To exclude files not relevant to the build, without restructuring your source -repository, use a `.dockerignore` file. This file supports exclusion patterns -similar to `.gitignore` files. For information on creating one, see -[.dockerignore file](../../engine/reference/builder.md#dockerignore-file). - ## Don't install unnecessary packages Avoid installing extra or unnecessary packages just because they might be nice to have. For example, you don’t need to include a text editor in a database image. @@ -48,23 +61,9 @@ Use your best judgment to keep containers as clean and modular as possible. If containers depend on each other, you can use [Docker container networks](../../network/index.md) to ensure that these containers can communicate. -## Minimize the number of layers - -In older versions of Docker, it was important that you minimized the number of -layers in your images to ensure they were performant. The following features -were added to reduce this limitation: - -- Only the instructions `RUN`, `COPY`, and `ADD` create layers. Other instructions - create temporary intermediate images, and don't increase the size of the build. - -- Where possible, use [multi-stage builds](../../build/building/multi-stage.md), - and only copy the artifacts you need into the final image. This allows you to - include tools and debug information in your intermediate build stages without - increasing the size of the final image. - ## Sort multi-line arguments -Whenever possible, sort multi-line arguments alphanumerically to make maintenance easier. +Whenever possible, sort multi-line arguments alphanumerically to make maintenance easier. This helps to avoid duplication of packages and make the list much easier to update. This also makes PRs a lot easier to read and review. Adding a space before a backslash (`\`) helps as well. @@ -81,213 +80,13 @@ RUN apt-get update && apt-get install -y \ && rm -rf /var/lib/apt/lists/* ``` -## Understand build context - -See [Build context](../../build/building/context.md) for more information. - -## Pipe a Dockerfile through stdin - -Docker has the ability to build images by piping a Dockerfile through stdin -with a local or remote build context. Piping a Dockerfile through stdin -can be useful to perform one-off builds without writing a Dockerfile to disk, -or in situations where the Dockerfile is generated, and should not persist -afterward. - -> **Note** -> -> The examples in the following sections use [here documents](https://tldp.org/LDP/abs/html/here-docs.html) -> for convenience, but any method to provide the Dockerfile on stdin can be -> used. -> -> For example, the following commands are equal: -> -> ```bash -> echo -e 'FROM busybox\nRUN echo "hello world"' | docker build - -> ``` -> -> ```bash -> docker build -< FROM busybox -> RUN echo "hello world" -> EOF -> ``` -> -> You can substitute the examples with your preferred approach, or the approach -> that best fits your use case. - -### Build an image using a Dockerfile from stdin, without sending build context - -Use this syntax to build an image using a Dockerfile from stdin, without -sending additional files as build context. The hyphen (`-`) takes the position -of the `PATH`, and instructs Docker to read the build context, which only -contains a Dockerfile, from stdin instead of a directory: - -```bash -docker build [OPTIONS] - -``` - -The following example builds an image using a Dockerfile that is passed through -stdin. No files are sent as build context to the daemon. - -```bash -docker build -t myimage:latest -< **Note** -> -> If you attempt to build an image using a Dockerfile from stdin, without sending a build context, then the build will fail if you use `COPY` or `ADD`. -> The following example illustrates this: -> -> ```bash -> # create a directory to work in -> mkdir example -> cd example -> -> # create an example file -> touch somefile.txt -> -> docker build -t myimage:latest -< FROM busybox -> COPY somefile.txt ./ -> RUN cat /somefile.txt -> EOF -> -> # observe that the build fails -> ... -> Step 2/3 : COPY somefile.txt ./ -> COPY failed: stat /var/lib/docker/tmp/docker-builder249218248/somefile.txt: no such file or directory -> ``` - -### Build from a local build context, using a Dockerfile from stdin - -Use this syntax to build an image using files on your local filesystem, but using -a Dockerfile from stdin. The syntax uses the `-f` (or `--file`) option to -specify the Dockerfile to use, and it uses a hyphen (`-`) as filename to instruct -Docker to read the Dockerfile from stdin: - -```bash -docker build [OPTIONS] -f- PATH -``` - -The following example uses the current directory (`.`) as the build context, and builds -an image using a Dockerfile that is passed through stdin using a [here -document](https://tldp.org/LDP/abs/html/here-docs.html). - -```bash -# create a directory to work in -mkdir example -cd example - -# create an example file -touch somefile.txt - -# build an image using the current directory as context, and a Dockerfile passed through stdin -docker build -t myimage:latest -f- . < **Note** -> -> When building an image using a remote Git repository as the build context, Docker -> performs a `git clone` of the repository on the local machine, and sends -> those files as the build context to the daemon. This feature requires you to -> install Git on the host where you run the `docker build` command. - -## Use multi-stage builds - -[Multi-stage builds](../../build/building/multi-stage.md) allow you to -drastically reduce the size of your final image, without struggling to reduce -the number of intermediate layers and files. - -Because an image is built during the final stage of the build process, you can -minimize image layers by [leveraging build cache](#leverage-build-cache). - -For example, if your build contains several layers and you want to ensure the build cache is reusable, you can order them from the less frequently changed to the more frequently changed. The following list is an example of the order of instructions: - -1. Install tools you need to build your application - -2. Install or update library dependencies - -3. Generate your application - -A Dockerfile for a Go application could look like: - -```dockerfile -# syntax=docker/dockerfile:1 -FROM golang:{{% param "example_go_version" %}}-alpine AS build - -# Install tools required for project -# Run `docker build --no-cache .` to update dependencies -RUN apk add --no-cache git - -# List project dependencies with go.mod and go.sum -# These layers are only re-built when Gopkg files are updated -WORKDIR /go/src/project/ -COPY go.mod go.sum /go/src/project/ -# Install library dependencies -RUN go mod download - -# Copy the entire project and build it -# This layer is rebuilt when a file changes in the project directory -COPY . /go/src/project/ -RUN go build -o /bin/project - -# This results in a single layer image -FROM scratch -COPY --from=build /bin/project /bin/project -ENTRYPOINT ["/bin/project"] -CMD ["--help"] -``` - ### Leverage build cache When building an image, Docker steps through the instructions in your -Dockerfile, executing each in the order specified. As each instruction is -examined, Docker looks for an existing image in its cache, -rather than creating a new, duplicate image. +Dockerfile, executing each in the order specified. For each instruction, Docker +checks whether it can reuse the instruction from the build cache. -If you don't want to use the cache at all, you can use the `--no-cache=true` -option on the `docker build` command. However, if you do let Docker use its -cache, it's important to understand when it can, and can't, find a matching -image. The basic rules that Docker follows are outlined below: +The basic rules of build cache invalidation are as follows: - Starting with a parent image that's already in the cache, the next instruction is compared against all child images derived from that base @@ -298,12 +97,10 @@ image. The basic rules that Docker follows are outlined below: of the child images is sufficient. However, certain instructions require more examination and explanation. -- For the `ADD` and `COPY` instructions, the contents of each file - in the image are examined and a checksum is calculated for each file. - The last-modified and last-accessed times of each file aren't considered in - these checksums. During the cache lookup, the checksum is compared against the - checksum in the existing images. If anything has changed in any file, such - as the contents and metadata, then the cache is invalidated. +- For the `ADD` and `COPY` instructions, the modification time and size file + metadata is used to determine whether cache is valid. During cache lookup, + cache is invalidated if the file metadata has changed for any of the files + involved. - Aside from the `ADD` and `COPY` commands, cache checking doesn't look at the files in the container to determine a cache match. For example, when processing @@ -313,3 +110,10 @@ image. The basic rules that Docker follows are outlined below: Once the cache is invalidated, all subsequent Dockerfile commands generate new images and the cache isn't used. + +If your build contains several layers and you want to ensure the build cache is +reusable, order the instructions from less frequently changed to more +frequently changed where possible. + +For more information about the Docker build cache and how to optimize your +builds, see [cache management](../../build/cache/_index.md). diff --git a/content/get-started/09_image_best.md b/content/get-started/09_image_best.md index 5510c85438..ed40052110 100644 --- a/content/get-started/09_image_best.md +++ b/content/get-started/09_image_best.md @@ -92,7 +92,7 @@ dependencies if there was a change to the `package.json`. `.dockerignore` files are an easy way to selectively copy only image relevant files. You can read more about this - [here](../engine/reference/builder.md#dockerignore-file). + [here](../build/building/context.md#dockerignore-files). In this case, the `node_modules` folder should be omitted in the second `COPY` step because otherwise, it would possibly overwrite files which were created by the command in the `RUN` step. For further details on why this is recommended for Node.js applications and other best practices, @@ -212,7 +212,7 @@ into an nginx container. In this section, you learned a few image building best practices, including layer caching and multi-stage builds. Related information: - - [.dockerignore](../engine/reference/builder.md#dockerignore-file) + - [.dockerignore](../build/building/context.md#dockerignore-files) - [Dockerfile reference](../engine/reference/builder.md) - [Build with Docker guide](../build/guide/index.md) - [Dockerfile best practices](../develop/develop-images/dockerfile_best-practices.md) diff --git a/content/network/proxy.md b/content/network/proxy.md index 157cf2d7e4..5d9b392443 100644 --- a/content/network/proxy.md +++ b/content/network/proxy.md @@ -188,4 +188,4 @@ If the proxy is an internal proxy, it might not be accessible for containers created from that image. Embedding proxy settings in images also poses a security risk, as the values -may include sensitive information. \ No newline at end of file +may include sensitive information. diff --git a/data/toc.yaml b/data/toc.yaml index 547a4972eb..d86ca1f28e 100644 --- a/data/toc.yaml +++ b/data/toc.yaml @@ -1784,7 +1784,7 @@ Manuals: - path: /build/building/packaging/ title: Packaging your software - path: /build/building/context/ - title: Build context + title: Context - path: /build/building/multi-stage/ title: Multi-stage builds - path: /build/building/multi-platform/