diff --git a/_config.yml b/_config.yml index a0eb6a01bd..361145f38a 100644 --- a/_config.yml +++ b/_config.yml @@ -190,7 +190,7 @@ fetch-remote: - repo: "https://github.com/moby/buildkit" default_branch: "master" - ref: "8bc51649bc7e712a8e2ff90412f48fdc35f602a9" + ref: "master" paths: - dest: "engine/reference/builder.md" src: diff --git a/_data/toc.yaml b/_data/toc.yaml index d65db16266..70f6b42ecd 100644 --- a/_data/toc.yaml +++ b/_data/toc.yaml @@ -1543,6 +1543,12 @@ manuals: title: Build contexts and linking targets - path: /build/customize/bake/compose-file/ title: Building from Compose file + - sectiontitle: BuildKit + section: + - path: /build/buildkit/ + title: Overview + - path: /build/buildkit/dockerfile-frontend/ + title: Custom Dockerfile syntax - sectiontitle: Buildx section: - path: /build/buildx/install/ diff --git a/build/buildkit/dockerfile-frontend.md b/build/buildkit/dockerfile-frontend.md new file mode 100644 index 0000000000..42d6b080c0 --- /dev/null +++ b/build/buildkit/dockerfile-frontend.md @@ -0,0 +1,102 @@ +--- +title: Custom Dockerfile syntax +keywords: build, buildkit, dockerfile, frontend +--- + +## Dockerfile frontend + +BuildKit supports loading frontends dynamically from container images. To use +an external Dockerfile frontend, the first line of your [Dockerfile](../../engine/reference/builder.md) +needs to set the [`syntax` directive](../../engine/reference/builder.md#syntax) +pointing to the specific image you want to use: + +```dockerfile +# syntax=[remote image reference] +``` + +For example: + +```dockerfile +# syntax=docker/dockerfile:1 +# syntax=docker.io/docker/dockerfile:1 +# syntax=example.com/user/repo:tag@sha256:abcdef... +``` + +This defines the location of the Dockerfile syntax that is used to build the +Dockerfile. The BuildKit backend allows seamlessly using external +implementations that are distributed as Docker images and execute inside a +container sandbox environment. + +Custom Dockerfile implementations allow you to: + +- Automatically get bugfixes without updating the Docker daemon +- Make sure all users are using the same implementation to build your Dockerfile +- Use the latest features without updating the Docker daemon +- Try out new features or third-party features before they are integrated in the Docker daemon +- Use [alternative build definitions, or create your own](https://github.com/moby/buildkit#exploring-llb){:target="_blank" rel="noopener" class="_"} + +> **Note** +> +> BuildKit also ships with a built-in Dockerfile frontend, but it's recommended +> to use an external image to make sure that all users use the same version on +> the builder and to pick up bugfixes automatically without waiting for a new +> version of BuildKit or Docker Engine. + +## Official releases + +Docker distributes official versions of the images that can be used for building +Dockerfiles under `docker/dockerfile` repository on Docker Hub. There are two +channels where new images are released: `stable` and `labs`. + +### Stable channel + +The `stable` channel follows [semantic versioning](https://semver.org){:target="_blank" rel="noopener" class="_"}. +For example: + +- `docker/dockerfile:1` - kept updated with the latest `1.x.x` minor _and_ patch + release. +- `docker/dockerfile:1.2` - kept updated with the latest `1.2.x` patch release, + and stops receiving updates once version `1.3.0` is released. +- `docker/dockerfile:1.2.1` - immutable: never updated. + +We recommend using `docker/dockerfile:1`, which always points to the latest +stable release of the version 1 syntax, and receives both "minor" and "patch" +updates for the version 1 release cycle. BuildKit automatically checks for +updates of the syntax when performing a build, making sure you are using the +most current version. + +If a specific version is used, such as `1.2` or `1.2.1`, the Dockerfile needs +to be updated manually to continue receiving bugfixes and new features. Old +versions of the Dockerfile remain compatible with the new versions of the +builder. + +### Labs channel + +The `labs` channel provides early access to Dockerfile features that are not yet +available in the `stable` channel. `labs` images are released at the same time +as stable releases, and follow the same version pattern, but use the `-labs` +suffix, for example: + +- `docker/dockerfile:labs` - latest release on `labs` channel. +- `docker/dockerfile:1-labs` - same as `dockerfile:1`, with experimental + features enabled. +- `docker/dockerfile:1.2-labs` - same as `dockerfile:1.2`, with experimental + features enabled. +- `docker/dockerfile:1.2.1-labs` - immutable: never updated. Same as + `dockerfile:1.2.1`, with experimental features enabled. + +Choose a channel that best fits your needs. If you want to benefit from +new features, use the `labs` channel. Images in the `labs` channel contain +all the features in the `stable` channel, plus early access features. +Stable features in the `labs` channel follow +[semantic versioning](https://semver.org){:target="_blank" rel="noopener" class="_"}, +but early access features don't, and newer releases may not be backwards compatible. +Pin the version to avoid having to deal with breaking changes. + +## Other resources + +For documentation on "labs" features, master builds, and nightly feature +releases, refer to the description in [the BuildKit source repository on GitHub](https://github.com/moby/buildkit/blob/master/README.md){:target="_blank" rel="noopener" class="_"}. +For a full list of available images, visit the [`docker/dockerfile` repository on Docker Hub](https://hub.docker.com/r/docker/dockerfile){:target="_blank" rel="noopener" class="_"}, +and the [`docker/dockerfile-upstream` repository on Docker Hub](https://hub.docker.com/r/docker/dockerfile-upstream){:target="_blank" rel="noopener" class="_"} +for development builds. diff --git a/build/buildkit/index.md b/build/buildkit/index.md new file mode 100644 index 0000000000..26879ad5e8 --- /dev/null +++ b/build/buildkit/index.md @@ -0,0 +1,65 @@ +--- +title: BuildKit +description: Introduction and overview of BuildKit +keywords: build, buildkit +--- + +## Overview + +[BuildKit](https://github.com/moby/buildkit){:target="_blank" rel="noopener" class="_"} +is an improved backend to replace the legacy builder. It comes with new and much +improved functionality for improving your builds' performance and the +reusability of your Dockerfiles. It also introduces support for handling more +complex scenarios: + +- Detect and skip executing unused build stages +- Parallelize building independent build stages +- Incrementally transfer only the changed files in your build context between builds +- Detect and skip transferring unused files in your build context +- Use [Dockerfile frontend](dockerfile-frontend.md) implementations with many new features +- Avoid side effects with rest of the API (intermediate images and containers) +- Prioritize your build cache for automatic pruning + +Apart from many new features, the main areas BuildKit improves on the current +experience are performance, storage management, and extensibility. From the +performance side, a significant update is a new fully concurrent build graph +solver. It can run build steps in parallel when possible and optimize out +commands that don't have an impact on the final result. We have also optimized +the access to the local source files. By tracking only the updates made to these +files between repeated build invocations, there is no need to wait for local +files to be read or uploaded before the work can begin. + +## LLB + +At the core of BuildKit is a [Low-Level Build (LLB)](https://github.com/moby/buildkit#exploring-llb){:target="_blank" rel="noopener" class="_"} +definition format. LLB is an intermediate binary format that allows developers +to extend BuildKit. LLB defines a content-addressable dependency graph that can +be used to put together very complex build definitions. It also supports +features not exposed in Dockerfiles, like direct data mounting and nested +invocation. + +![Directed acyclic graph (DAG)](../images/buildkit-dag.svg){:class="invertible" style="width:60%"} + +Everything about execution and caching of your builds is defined in LLB. The +caching model is entirely rewritten compared to the legacy builder. Rather than +using heuristics to compare images, LLB directly tracks the checksums of build +graphs and content mounted to specific operations. This makes it much faster, +more precise, and portable. The build cache can even be exported to a registry, +where it can be pulled on-demand by subsequent invocations on any host. + +LLB can be generated directly using a [golang client package](https://pkg.go.dev/github.com/moby/buildkit/client/llb) +that allows defining the relationships between your build operations using Go +language primitives. This gives you full power to run anything you can imagine, +but will probably not be how most people will define their builds. Instead, +most users would use a frontend component, or LLB nested invocation, to run +a prepared set of build steps. + +## Frontend + +A frontend is a component that takes a human-readable +build format and converts it to LLB so BuildKit can execute it. Frontends can +be distributed as images, and the user can target a specific version of a +frontend that is guaranteed to work for the features used by their definition. + +For example, to build a [Dockerfile](../../engine/reference/builder.md) with +BuildKit, you would [use an external Dockerfile frontend](dockerfile-frontend.md). diff --git a/build/images/buildkit-dag.svg b/build/images/buildkit-dag.svg new file mode 100644 index 0000000000..ead20f9d9e --- /dev/null +++ b/build/images/buildkit-dag.svg @@ -0,0 +1,74 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/build/index.md b/build/index.md index 2d4f917864..195eeaa916 100644 --- a/build/index.md +++ b/build/index.md @@ -20,29 +20,24 @@ and tools. The most common method of executing a build is by issuing a sends the request to Docker Engine which, in turn, executes your build. There are now two components in Engine that can be used to build an image. -Starting with the [18.09 release](../engine/release-notes/18.09.md#18090), Engine is -shipped with Moby [BuildKit](https://github.com/moby/buildkit){:target="_blank" rel="noopener" class="_"}, -the new component for executing your builds by default. - -BuildKit is the backend evolution from the Legacy Builder, it comes with new -and much improved functionality that can be powerful tools for improving your -builds' performance or reusability of your Dockerfiles, and it also introduces -support for complex scenarios. +Starting with the [18.09 release](../engine/release-notes/18.09.md#18090), +Engine is shipped with Moby [BuildKit](buildkit/index.md), the new component for +executing your builds by default. The new client [Docker Buildx](https://github.com/docker/buildx){:target="_blank" rel="noopener" class="_"}, is a CLI plugin that extends the docker command with the full support of the -features provided by BuildKit builder toolkit. `docker buildx build` provides -the same user experience as `docker build` with many new features like creating -scoped builder instances, building against multiple nodes concurrently, outputs -configuration, inline build caching, and specifying target platform. In -addition, Buildx also supports new features that are not yet available for -regular `docker build` like building manifest lists, distributed caching, and -exporting build results to OCI image tarballs. +features provided by [BuildKit](buildkit/index.md) builder toolkit. [`docker buildx build` command](../engine/reference/commandline/buildx_build.md) +provides the same user experience as `docker build` with many new features like +creating scoped [builder instances](building/drivers/index.md), building +against multiple nodes concurrently, outputs configuration, inline [build caching](building/cache/index.md), +and specifying target platform. In addition, Buildx also supports new features +that are not yet available for regular `docker build` like building manifest +lists, distributed caching, and exporting build results to OCI image tarballs. Docker Build is way more than a simple build command and is not only about packaging your code, it's a whole ecosystem of tools and features that support not only common workflow tasks but also provides support for more complex and -advanced scenarios: +advanced scenarios. ## Building your images @@ -114,13 +109,14 @@ to be built concurrently as part of a single request: [High-level builds with Bake](customize/bake/index.md){: .button .outline-btn } -## Extending BuildKit +## BuildKit -### Custom syntax on Dockerfile +### Custom Dockerfile syntax Use experimental versions of the Dockerfile frontend, or even just bring your -own to BuildKit using the power of custom frontends. See also the -[Syntax directive](../engine/reference/builder.md#syntax). +own to BuildKit using the power of custom frontends. + +[Custom Dockerfile syntax](buildkit/dockerfile-frontend.md){: .button .outline-btn } ### Configure BuildKit