mirror of https://github.com/docker/docs.git
454 lines
16 KiB
Markdown
454 lines
16 KiB
Markdown
---
|
||
title: Hydrobuild
|
||
description: Get started with Docker Hydrobuild
|
||
sitemap: false
|
||
---
|
||
|
||
> **Early Access**
|
||
>
|
||
> Docker Hydrobuild is an early-access service that provides cloud-based
|
||
> builders for your Docker organization.
|
||
>
|
||
> If you want to get involved in testing Hydrobuild, you can
|
||
> [sign up for the early access program](https://www.docker.com/build-early-access-program/?utm_source=docs).
|
||
{ .restricted }
|
||
|
||
Hydrobuild is a service that lets you build your container images faster, both
|
||
locally and in CI. Builds run on cloud infrastructure optimally dimensioned for
|
||
your workloads, no configuration required. The service uses a remote build
|
||
cache, ensuring fast builds anywhere and for all team members.
|
||
|
||
## How Hydrobuild works
|
||
|
||
Using Hydrobuild is no different from running a regular build. You invoke a
|
||
build the same way you normally would, using `docker buildx build`. The
|
||
difference is in where and how that build gets executed.
|
||
|
||
By default when you invoke a build command, the build runs on a local instance
|
||
of BuildKit, bundled with the Docker daemon. With Hydrobuild, you send the
|
||
build request to a BuildKit instance running remotely, in the cloud.
|
||
|
||
The remote builder executes the build steps, and sends the resulting build
|
||
output to the destination that you specify. For example, back to your local
|
||
Docker Engine image store, or to an image registry.
|
||
|
||
Hydrobuild provides several benefits over local builds:
|
||
|
||
- Improved build speed
|
||
- Shared build cache
|
||
- Native multi-platform builds
|
||
|
||
And the best part: you don't need to worry about managing builders or
|
||
infrastructure. Just connect to your builders, and start building.
|
||
|
||
## Setup
|
||
|
||
To get started with Hydrobuild, you need to:
|
||
|
||
- Download and install a version of Buildx that supports Hydrobuild.
|
||
|
||
- Have a Docker ID that's part of a Docker organization participating in the
|
||
[Hydrobuild early access program](https://www.docker.com/build-early-access-program/?utm_source=docs).
|
||
|
||
Docker Desktop 4.23.0 and later versions ship with a Hydrobuild-compatible
|
||
Buildx binary. Alternatively, you can download and install the binary manually
|
||
from [this repository](https://github.com/docker/buildx-desktop).
|
||
|
||
## Connecting to Hydrobuild
|
||
|
||
To start building with Hydrobuild, you must create a new builder using the
|
||
`docker buildx create` command. The builder is connected to Hydrobuild through
|
||
an endpoint that you specify. The endpoint represents a single, isolated
|
||
builder. Builder endpoints have the following format:
|
||
|
||
cloud://<org>/default
|
||
|
||
`<org>` is the Docker organization that the builder is provisioned for.
|
||
|
||
The builders have native support for the `linux/amd64` and `linux/arm64`
|
||
architectures by default. This gives you a high-performance build cluster for
|
||
building multi-platform images natively.
|
||
|
||
You can omit the `cloud://` protocol prefix from the endpoint when you create a
|
||
builder using the `cloud` driver.
|
||
|
||
### Create a builder
|
||
|
||
1. Sign in to your Docker ID using the Docker Desktop UI or the `docker login`
|
||
command.
|
||
|
||
2. Create a builder that uses the `cloud` driver.
|
||
|
||
```console
|
||
$ docker buildx create --driver cloud <org>/default
|
||
```
|
||
|
||
Replace `<org>` with the Docker organization.
|
||
|
||
This creates a builder named `cloud-<org>-default`.
|
||
|
||
## Use Hydrobuild from the CLI
|
||
|
||
To run a build using Hydrobuild, invoke a build command and specify the
|
||
name of the builder using the `--builder` flag.
|
||
|
||
```console
|
||
$ docker buildx build --builder cloud-<org>-default --tag <org>/<image> .
|
||
```
|
||
|
||
> **Note**
|
||
>
|
||
> Building with `--tag` loads the build result to the local image store
|
||
> automatically when the build finishes. To build without a tag and load the
|
||
> result, you must pass the `--load` flag.
|
||
>
|
||
> If you use the containerd image store, you must always pass `--load` to
|
||
> download the results, even if you build with a tag.
|
||
|
||
To build a multi-platform image and push it to a registry:
|
||
|
||
```console
|
||
$ docker buildx build --builder cloud-<org>-default \
|
||
--platform linux/amd64,linux/arm64 \
|
||
--tag <org>/<image> \
|
||
--push .
|
||
```
|
||
|
||
> **Note**
|
||
>
|
||
> If you build multi-platform images, you won't be able to load the images back
|
||
> to your local image store unless you turn on the containerd image store
|
||
> feature, and use the `--load` flag.
|
||
>
|
||
> Using the containerd image store with Hydrobuild currently results in slower
|
||
> transfers of build output to the client, compared to when you use the default
|
||
> image store.
|
||
>
|
||
> When building multi-platform images, consider pushing the resulting image to
|
||
> a registry directly, using the `docker buildx build --push` flag.
|
||
|
||
### Use by default
|
||
|
||
If you want to use Hydrobuild by default, you can run the following command to
|
||
make it the selected builder:
|
||
|
||
```console
|
||
$ docker buildx use cloud-<org>-default --global
|
||
```
|
||
|
||
> **Note**
|
||
>
|
||
> Changing your default builder with `docker buildx use` only changes the
|
||
> default builder for the `docker buildx build` command. The `docker build`
|
||
> command still uses the `default` builder, unless you specify the `--builder`
|
||
> flag explicitly.
|
||
>
|
||
> If you use build scripts, such as `make`, we recommend that you update your
|
||
> build commands from `docker build` to `docker buildx build`, to avoid any
|
||
> confusion with regards to builder selection. Alternatively, you can run
|
||
> `docker buildx install` to make the default `docker build` command behave
|
||
> like `docker buildx build`, without discrepancies.
|
||
|
||
## Use Hydrobuild in CI
|
||
|
||
Using Hydrobuild in CI can speed up your build pipelines, which means less time
|
||
spent waiting and context switching. You control your CI workflows as usual,
|
||
and delegate the build execution to Hydrobuild.
|
||
|
||
Building with Hydrobuild in CI involve the following steps:
|
||
|
||
1. Sign in to a Docker account.
|
||
2. Set up Buildx and create the builder.
|
||
3. Run the build.
|
||
|
||
When using Hydrobuild in CI, it's recommended that you push the result to a
|
||
registry directly, rather than loading the image and then pushing it. Pushing
|
||
directly speeds up your builds and avoids unnecessary file transfers.
|
||
|
||
If you just want to build and discard the output, export the results to the
|
||
build cache using `--output type=cacheonly`, or build without a `tag`.
|
||
Hydrobuild automatically loads the build result if you build a tagged image.
|
||
|
||
{{< tabs >}}
|
||
{{< tab name="GitHub Actions" >}}
|
||
|
||
```yaml
|
||
name: ci
|
||
|
||
on:
|
||
push:
|
||
branches:
|
||
- "main"
|
||
|
||
jobs:
|
||
docker:
|
||
runs-on: ubuntu-latest
|
||
steps:
|
||
- name: Checkout
|
||
uses: actions/checkout@v4
|
||
- name: Log in to Docker Hub
|
||
uses: docker/login-action@v3
|
||
with:
|
||
username: ${{ secrets.DOCKERHUB_USERNAME }}
|
||
password: ${{ secrets.DOCKERHUB_TOKEN }}
|
||
- name: Set up Docker Buildx
|
||
uses: docker/setup-buildx-action@v3
|
||
with:
|
||
version: "lab:latest"
|
||
driver: cloud
|
||
endpoint: "<org>/default"
|
||
- name: Build and push
|
||
uses: docker/build-push-action@v5
|
||
with:
|
||
context: .
|
||
tags: "<org>/<image>"
|
||
# For pull requests, export results to the build cache.
|
||
# Otherwise, push to a registry.
|
||
outputs: ${{ github.event_name == 'pull_request' && 'type=cacheonly' || 'type=registry,push=true' }}
|
||
```
|
||
|
||
{{< /tab >}}
|
||
{{< tab name="CircleCI" >}}
|
||
|
||
```yaml
|
||
version: 2.1
|
||
|
||
jobs:
|
||
# Build multi-platform image and push to a registry
|
||
build_push:
|
||
machine:
|
||
image: ubuntu-2204:current
|
||
steps:
|
||
- checkout
|
||
|
||
- run: |
|
||
mkdir -vp ~/.docker/cli-plugins/
|
||
ARCH=amd64
|
||
BUILDX_URL=$(curl -s https://raw.githubusercontent.com/docker/actions-toolkit/main/.github/buildx-lab-releases.json | jq -r ".latest.assets[] | select(endswith(\"linux-$ARCH\"))")
|
||
curl --silent -L --output ~/.docker/cli-plugins/docker-buildx $BUILDX_URL
|
||
chmod a+x ~/.docker/cli-plugins/docker-buildx
|
||
|
||
- run: echo "$DOCKER_PASS" | docker login --username $DOCKER_USER --password-stdin
|
||
- run: docker buildx create --use --driver cloud "<org>/default"
|
||
|
||
- run: |
|
||
docker buildx build \
|
||
--platform linux/amd64,linux/arm64 \
|
||
--push \
|
||
--tag "<org>/<image>" .
|
||
|
||
# Build an image and discard the result
|
||
build_cache:
|
||
machine:
|
||
image: ubuntu-2204:current
|
||
steps:
|
||
- checkout
|
||
|
||
- run: |
|
||
mkdir -vp ~/.docker/cli-plugins/
|
||
ARCH=amd64
|
||
BUILDX_URL=$(curl -s https://raw.githubusercontent.com/docker/actions-toolkit/main/.github/buildx-lab-releases.json | jq -r ".latest.assets[] | select(endswith(\"linux-$ARCH\"))")
|
||
curl --silent -L --output ~/.docker/cli-plugins/docker-buildx $BUILDX_URL
|
||
chmod a+x ~/.docker/cli-plugins/docker-buildx
|
||
|
||
- run: echo "$DOCKER_PASS" | docker login --username $DOCKER_USER --password-stdin
|
||
- run: docker buildx create --use --driver cloud "<org>/default"
|
||
|
||
- run: |
|
||
docker buildx build \
|
||
--tag temp \
|
||
--output type=cacheonly \
|
||
.
|
||
|
||
workflows:
|
||
pull_request:
|
||
jobs:
|
||
- build_cache
|
||
release:
|
||
jobs:
|
||
- build_push
|
||
```
|
||
|
||
{{< /tab >}}
|
||
{{< tab name="Shell" >}}
|
||
|
||
```bash
|
||
#!/bin/bash
|
||
|
||
# Get download link for latest buildx binary. Set $ARCH to the CPU architecture (e.g. amd64, arm64)
|
||
ARCH=amd64
|
||
BUILDX_URL=$(curl -s https://raw.githubusercontent.com/docker/actions-toolkit/main/.github/buildx-lab-releases.json | jq -r ".latest.assets[] | select(endswith(\"linux-$ARCH\"))")
|
||
|
||
# Download docker buildx with Hyrdobuild support
|
||
mkdir -vp ~/.docker/cli-plugins/
|
||
curl --silent -L --output ~/.docker/cli-plugins/docker-buildx $BUILDX_URL
|
||
chmod a+x ~/.docker/cli-plugins/docker-buildx
|
||
|
||
# Login to Docker Hub. For security reasons $DOCKER_PASS should be a Personal Access Token. See https://docs.docker.com/docker-hub/access-tokens/
|
||
echo "$DOCKER_PASS" | docker login --username $DOCKER_USER --password-stdin
|
||
|
||
# Connect to your builder and set it as the default builder
|
||
docker buildx create --use --driver cloud "<org>/default"
|
||
|
||
# Cache-only image build
|
||
docker buildx build \
|
||
--tag temp \
|
||
--output type=cacheonly \
|
||
.
|
||
|
||
# Build, tag, and push a multi-arch docker image
|
||
docker buildx build \
|
||
--platform linux/amd64,linux/arm64 \
|
||
--push \
|
||
--tag "<org>/<image>" \
|
||
.
|
||
```
|
||
|
||
{{< /tab >}}
|
||
{{< /tabs >}}
|
||
|
||
## Hydrobuild in Docker Desktop
|
||
|
||
The Docker Desktop [Builds view](../desktop/use-desktop/builds.md) works with
|
||
Hydrobuild out of the box. With Hydrobuild, the Builds view becomes a
|
||
collaboration tool, showing information about not only your own builds, but
|
||
also builds initiated by your team members using the same builder.
|
||
|
||
Teams using a shared builder get access to information such as:
|
||
|
||
- Ongoing and completed builds
|
||
- Build configuration, statistics, dependencies, and results
|
||
- Build source (Dockerfile)
|
||
- Build logs and errors
|
||
|
||
This lets you and your team can work collaboratively on troubleshooting and
|
||
improving build speeds, without having to send build logs and benchmarks back
|
||
and forth between each other.
|
||
|
||
## Optimize for building in the cloud
|
||
|
||
Hydrobuild runs your builds remotely, and not on the machine where you invoke
|
||
the build. This means that file transfers between the client and builder happens
|
||
over the network.
|
||
|
||
Transferring files over the network has a higher latency and lower bandwidth
|
||
than local transfers. Hydrobuild has several features to mitigate this:
|
||
|
||
- It uses attached storage volumes for build cache, which makes reading and
|
||
writing cache very fast.
|
||
- Loading build results back to the client only pulls the layers that were
|
||
changed compared to previous builds.
|
||
|
||
Despite these optimizations, building remotely can still yield slow context
|
||
transfers and image loads, for large projects or if the network connection is
|
||
slow. Here are some ways that you can optimize your builds to make the transfer
|
||
more efficient:
|
||
|
||
- [Dockerignore files](#dockerignore-files)
|
||
- [Slim base images](#slim-base-images)
|
||
- [Multi-stage builds](#multi-stage-builds)
|
||
- [Fetch remote files in build](#fetch-remote-files-in-build)
|
||
- [Multi-threaded tools](#multi-threaded-tools)
|
||
|
||
### Dockerignore files
|
||
|
||
Using a [`.dockerignore` file](./building/context.md#dockerignore), you can be
|
||
explicit about which local files that you don’t want to include in the build
|
||
context. Files caught by the [glob patterns](../engine/reference/builder.md#dockerignore-file)
|
||
you specify in your ignore-file are not transferred to the remote builder.
|
||
|
||
Some examples of things you might want to add to your `.dockerignore` file are:
|
||
|
||
- `.git` — skip sending the version control history in the build context. Note
|
||
that this means you won’t be able to run Git commands in your build steps,
|
||
such as `git rev-parse` etc.
|
||
- Directories containing build artifacts, such as binaries. Build artifacts
|
||
created locally during development.
|
||
- Vendor directories for package managers, such as `node_modules`.
|
||
|
||
In general, the contents of your `.dockerignore` file should be similar to what
|
||
you have in your `.gitignore`.
|
||
|
||
### Slim base images
|
||
|
||
Selecting smaller images for your `FROM` instructions in your Dockerfile can
|
||
help reduce the size of the final image. The [Alpine image](https://hub.docker.com/_/alpine)
|
||
is a good example of a minimal Docker image that provides all of the OS
|
||
utilities you would expect from a Linux container.
|
||
|
||
There’s also the [special `scratch` image](https://hub.docker.com/_/scratch),
|
||
which contains nothing at all. Useful for creating images of statically linked
|
||
binaries, for example.
|
||
|
||
### Multi-stage builds
|
||
|
||
[Multi-stage builds](./guide/multi-stage.md) can make your build run faster,
|
||
because stages can run in parallel. It can also make your end-result smaller.
|
||
Write your Dockerfile in such a way that the final runtime stage uses the
|
||
smallest possible base image, with only the resources that your program requires
|
||
to run.
|
||
|
||
It’s also possible to
|
||
[copy resources from other images or stages](./building/multi-stage.md#name-your-build-stages),
|
||
using the Dockerfile `COPY --from` instruction. This technique can reduce the
|
||
number of layers, and the size of those layers, in the final stage.
|
||
|
||
### Fetch remote files in build
|
||
|
||
When possible, you should fetch files from a remote location in the build,
|
||
rather than bundling the files into the build context. Downloading files on the
|
||
Hydrobuild server directly is better, because it will likely be faster than
|
||
transferring the files with the build context.
|
||
|
||
You can fetch remote files during the build using the
|
||
[Dockerfile `ADD` instruction](../engine/reference/builder.md#add),
|
||
or in your `RUN` instructions with tools like `wget` and `rsync`.
|
||
|
||
### Multi-threaded tools
|
||
|
||
Some tools that you use in your build instructions may not utilize multiple
|
||
cores by default. One such example is `make` which uses a single thread by
|
||
default, unless you specify the `make --jobs=<n>` option. For build steps
|
||
involving such tools, try checking if you can optimize the execution with
|
||
parallelization.
|
||
|
||
## Frequently asked questions
|
||
|
||
### How do I remove Hydrobuild from my system?
|
||
|
||
If you want to stop using Hydrobuild, and remove it from your system, remove
|
||
the builder using the `docker buildx rm` command.
|
||
|
||
```console
|
||
$ docker buildx rm cloud-<org>-default
|
||
```
|
||
|
||
This doesn't deprovision the builder backend, it only removes the builder from
|
||
your local Docker client.
|
||
|
||
### Are builders shared between organizations?
|
||
|
||
No. Each Hydrobuild builder provisioned to an organization is completely
|
||
isolated to a single Amazon EC2 instance, with a dedicated EBS volume for build
|
||
cache, and end-to-end encryption. That means there are no shared processes or
|
||
data between Hydrobuild jobs.
|
||
|
||
### Do I need to add my secrets the builder to access private resources?
|
||
|
||
No. Your interface to Hydrobuild is Buildx, and you can use the existing
|
||
`--secret` and `--ssh` CLI flags for managing build secrets.
|
||
|
||
For more information, refer to:
|
||
|
||
- [docker buildx build --secret](../engine/reference/commandline/buildx_build.md#secret)
|
||
- [docker buildx build --ssh](../engine/reference/commandline/buildx_build.md#ssh)
|
||
|
||
### How do I unset Hydrobuild as the default builder?
|
||
|
||
If you've set Hydrobuild as the default builder and want to revert to using the
|
||
default `docker` builder, run the following command:
|
||
|
||
```console
|
||
$ docker context use default
|
||
```
|