From 5e789ba152509df821f212a04be44a18ab5a0da1 Mon Sep 17 00:00:00 2001 From: Alejandro Pedraza Date: Wed, 22 Jul 2020 14:27:45 -0500 Subject: [PATCH] Migrate CI to docker buildx and other improvements (#4765) * Migrate CI to docker buildx and other improvements ## Motivation - Improve build times in forks. Specially when rerunning builds because of some flaky test. - Start using `docker buildx` to pave the way for multiplatform builds. ## Performance improvements These timings were taken for the `kind_integration.yml` workflow when we merged and rerun the lodash bump PR (#4762) Before these improvements: - when merging: `24:18` - when rerunning after merge (docker cache warm): `19:00` - when running the same changes in a fork (no docker cache): `32:15` After these improvements: - when merging: `25:38` - when rerunning after merge (docker cache warm): `19:25` - when running the same changes in a fork (docker cache warm): `19:25` As explained below, non-forks and forks now use the same cache, so the important take is that forks will always start with a warm cache and we'll no longer see long build times like the `32:15` above. The downside is a slight increase in the build times for non-forks (up to a little more than a minute, depending on the case). ## Build containers in parallel The `docker_build` job in the `kind_integration.yml`, `cloud_integration.yml` and `release.yml` workflows relied on running `bin/docker-build` which builds all the containers in sequence. Now each container is built in parallel using a matrix strategy. ## New caching strategy CI now uses `docker buildx` for building the container images, which allows using an external cache source for builds, a location in the filesystem in this case. That location gets cached using actions/cache, using the key `{{ runner.os }}-buildx-${{ matrix.target }}-${{ env.TAG }}` and the restore key `${{ runner.os }}-buildx-${{ matrix.target }}-`. For example when building the `web` container, its image and all the intermediary layers get cached under the key `Linux-buildx-web-git-abc0123`. When that has been cached in the `main` branch, that cache will be available to all the child branches, including forks. If a new branch in a fork asks for a key like `Linux-buildx-web-git-def456`, the key won't be found during the first CI run, but the system falls back to the key `Linux-buildx-web-git-abc0123` from `main` and so the build will start with a warm cache (more info about how keys are matched in the [actions/cache docs](https://docs.github.com/en/actions/configuring-and-managing-workflows/caching-dependencies-to-speed-up-workflows#matching-a-cache-key)). ## Packet host no longer needed To benefit from the warm caches both in non-forks and forks like just explained, we're required to ditch doing the builds in Packet and now everything runs in the github runners VMs. As a result there's no longer separate logic for non-forks and forks in the workflow files; `kind_integration.yml` was greatly simplified but `cloud_integration.yml` and `release.yml` got a little bigger in order to use the actions artifacts as a repository for the images built. This bloat will be fixed when support for [composite actions](https://github.com/actions/runner/blob/users/ethanchewy/compositeADR/docs/adrs/0549-composite-run-steps.md) lands in github. ## Local builds You still are able to run `bin/docker-build` or any of the `docker-build.*` scripts. And to make use of buildx, run those same scripts after having set the env var `DOCKER_BUILDKIT=1`. Using buildx supposes you have installed it, as instructed [here](https://github.com/docker/buildx). ## Other - A new script `bin/docker-cache-prune` is used to remove unused images from the cache. Without that the cache grows constantly and we can rapidly hit the 5GB limit (when the limit is attained the oldest entries get evicted). - The `go-deps` dockerfile base image was changed from `golang:1.14.2` (ubuntu based) to `golang-1:14.2-alpine` also to conserve cache space. # Addressed separately in #4875: Got rid of the `go-deps` image and instead added something similar on top of all the Dockerfiles dealing with `go`, as a first stage for those Dockerfiles. That continues to serve as a way to pre-populate go's build cache, which speeds up the builds in the subsequent stages. That build should in theory be rebuilt automatically only when `go.mod` or `go.sum` change, and now we don't require running `bin/update-go-deps-shas`. That script was removed along with all the logic elsewhere that used it, including the `go_dependencies` job in the `static_checks.yml` github workflow. The list of modules preinstalled was moved from `Dockerfile-go-deps` to a new script `bin/install-deps`. I couldn't find a way to generate that list dynamically, so whenever a slow-to-compile dependency is found, we have to make sure it's included in that list. Although this simplifies the dev workflow, note that the real motivation behind this was a limitation in buildx's `docker-container` driver that forbids us from depending on images that haven't been pushed to a registry, so we have to resort to building the dependencies as a first stage in the Dockerfiles. --- .dockerignore | 1 + .github/workflows/cloud_integration.yml | 94 +++++------ .github/workflows/kind_integration.yml | 75 +++------ .github/workflows/release.yml | 173 ++++++++------------- .github/workflows/static_checks.yml | 23 --- BUILD.md | 23 --- Dockerfile-proxy | 10 +- bin/_docker.sh | 33 +++- bin/_tag.sh | 36 ----- bin/docker-build-cli-bin | 7 - bin/docker-build-cni-plugin | 3 - bin/docker-build-controller | 7 - bin/docker-build-go-deps | 26 ---- bin/docker-build-proxy | 6 - bin/docker-build-web | 7 - bin/docker-cache-prune | 21 +++ bin/docker-images | 2 - bin/{docker-pull-deps => docker-pull-base} | 3 - bin/{docker-push-deps => docker-push-base} | 3 - Dockerfile-go-deps => bin/install-deps | 26 +--- bin/update-go-deps-shas | 13 -- cli/Dockerfile-bin | 10 +- cni-plugin/Dockerfile | 10 +- controller/Dockerfile | 10 +- web/Dockerfile | 10 +- 25 files changed, 237 insertions(+), 395 deletions(-) delete mode 100755 bin/docker-build-go-deps create mode 100755 bin/docker-cache-prune rename bin/{docker-pull-deps => docker-pull-base} (64%) rename bin/{docker-push-deps => docker-push-base} (65%) rename Dockerfile-go-deps => bin/install-deps (80%) mode change 100644 => 100755 delete mode 100755 bin/update-go-deps-shas diff --git a/.dockerignore b/.dockerignore index 92ece2ba8..ced00b0a0 100644 --- a/.dockerignore +++ b/.dockerignore @@ -7,6 +7,7 @@ **/node_modules bin !bin/fetch-proxy +!bin/install-deps !bin/web **/Dockerfile* Dockerfile* diff --git a/.github/workflows/cloud_integration.yml b/.github/workflows/cloud_integration.yml index 01f1991ae..7491d10a2 100644 --- a/.github/workflows/cloud_integration.yml +++ b/.github/workflows/cloud_integration.yml @@ -8,35 +8,14 @@ on: - main env: GH_ANNOTATION: true + DOCKER_BUILDKIT: 1 jobs: - # todo: Keep in sync with `release.yml` docker_build: - name: Docker build runs-on: ubuntu-18.04 - steps: - - name: Checkout code - # actions/checkout@v2 - uses: actions/checkout@722adc6 - - name: Setup SSH config for Packet - run: | - mkdir -p ~/.ssh/ - touch ~/.ssh/id && chmod 600 ~/.ssh/id - echo "${{ secrets.DOCKER_SSH_CONFIG }}" > ~/.ssh/config - echo "${{ secrets.DOCKER_PRIVATE_KEY }}" > ~/.ssh/id - echo "${{ secrets.DOCKER_KNOWN_HOSTS }}" > ~/.ssh/known_hosts - ssh linkerd-docker docker version - - name: Build docker images - env: - DOCKER_HOST: ssh://linkerd-docker - DOCKER_TRACE: 1 - run: | - export PATH="`pwd`/bin:$PATH" - bin/docker-build - # todo: Keep in sync with `release.yml` - docker_push: - name: Docker push - runs-on: ubuntu-18.04 - needs: [docker_build] + strategy: + matrix: + target: [proxy, controller, web, cni-plugin, debug, cli-bin, grafana] + name: Docker build (${{ matrix.target }}) steps: - name: Checkout code # actions/checkout@v2 @@ -45,6 +24,24 @@ jobs: run: | . bin/_tag.sh echo ::set-env name=TAG::$(CI_FORCE_CLEAN=1 bin/root-tag) + + . bin/_docker.sh + echo ::set-env name=DOCKER_REGISTRY::$DOCKER_REGISTRY + echo ::set-env name=DOCKER_BUILDKIT_CACHE::${{ runner.temp }}/.buildx-cache + - name: Cache docker layers + # actions/cache@v2.0.0 + uses: actions/cache@b820478 + with: + path: ${{ env.DOCKER_BUILDKIT_CACHE }} + key: ${{ runner.os }}-buildx-${{ matrix.target }}-${{ env.TAG }} + restore-keys: | + ${{ runner.os }}-buildx-${{ matrix.target }}- + - name: Build docker images + env: + DOCKER_TRACE: 1 + run: | + docker buildx create --driver docker-container --use + bin/docker-build-${{ matrix.target }} - name: Configure gcloud # linkerd/linkerd2-action-gcloud@v1.0.1 uses: linkerd/linkerd2-action-gcloud@308c4df @@ -52,28 +49,21 @@ jobs: cloud_sdk_service_account_key: ${{ secrets.CLOUD_SDK_SERVICE_ACCOUNT_KEY }} gcp_project: ${{ secrets.GCP_PROJECT }} gcp_zone: ${{ secrets.GCP_ZONE }} - - name: Docker SSH setup - run: | - mkdir -p ~/.ssh/ - touch ~/.ssh/id && chmod 600 ~/.ssh/id - echo "${{ secrets.DOCKER_SSH_CONFIG }}" > ~/.ssh/config - echo "${{ secrets.DOCKER_PRIVATE_KEY }}" > ~/.ssh/id - echo "${{ secrets.DOCKER_KNOWN_HOSTS }}" > ~/.ssh/known_hosts - ssh linkerd-docker docker version - name: Push docker images to registry - env: - DOCKER_HOST: ssh://linkerd-docker run: | - export PATH="`pwd`/bin:$PATH" - bin/docker-push-deps - bin/docker-push $TAG - bin/docker-retag-all $TAG main - bin/docker-push main + . bin/_docker.sh + docker_push "${{ matrix.target }}" "$TAG" + docker_retag "${{ matrix.target }}" "$TAG" main + docker_push "${{ matrix.target }}" main + - name: Prune docker layers cache + # changes generate new images while the existing ones don't get removed + # so we manually do that to avoid bloating the cache + run: bin/docker-cache-prune # todo: Keep in sync with `release.yml` cloud_integration_tests: name: Cloud integration tests runs-on: ubuntu-18.04 - needs: [docker_push] + needs: [docker_build] steps: - name: Checkout code # actions/checkout@v2 @@ -90,14 +80,14 @@ jobs: id: install_cli run: | TAG="$(CI_FORCE_CLEAN=1 bin/root-tag)" - image="gcr.io/linkerd-io/cli-bin:$TAG" - id=$(bin/docker create $image) - bin/docker cp "$id:/out/linkerd-linux" "$HOME/.linkerd" - "$HOME/.linkerd" version --client + CMD="$PWD/target/release/linkerd2-cli-$TAG-linux" + bin/docker-pull-binaries $TAG + $CMD version --client # validate CLI version matches the repo - [[ "$TAG" == "$($HOME/.linkerd version --short --client)" ]] + [[ "$TAG" == "$($CMD version --short --client)" ]] echo "Installed Linkerd CLI version: $TAG" - echo "::set-output name=tag::$TAG" + echo "::set-env name=CMD::$CMD" + echo "::set-env name=TAG::$TAG" - name: Create GKE cluster # linkerd/linkerd2-action-gcloud@v1.0.1 uses: linkerd/linkerd2-action-gcloud@308c4df @@ -107,17 +97,15 @@ jobs: gcp_zone: ${{ secrets.GCP_ZONE }} preemptible: false create: true - name: testing-${{ steps.install_cli.outputs.tag }}-${{ github.run_id }} + name: testing-${{ env.TAG }}-${{ github.run_id }} num_nodes: 2 - name: Run integration tests env: GITCOOKIE_SH: ${{ secrets.GITCOOKIE_SH }} run: | - export PATH="`pwd`/bin:$PATH" echo "$GITCOOKIE_SH" | bash - version="$($HOME/.linkerd version --client --short | tr -cd '[:alnum:]-')" - bin/tests --skip-kind-create "$HOME/.linkerd" + bin/tests --skip-kind-create "$CMD" - name: CNI tests run: | - export TAG="$($HOME/.linkerd version --client --short)" + export TAG="$($CMD version --client --short)" go test -cover -race -v -mod=readonly ./cni-plugin/test -integration-tests diff --git a/.github/workflows/kind_integration.yml b/.github/workflows/kind_integration.yml index 6ccd02e2d..fc82501b5 100644 --- a/.github/workflows/kind_integration.yml +++ b/.github/workflows/kind_integration.yml @@ -9,10 +9,14 @@ on: - main env: GH_ANNOTATION: true + DOCKER_BUILDKIT: 1 jobs: docker_build: - name: Docker build runs-on: ubuntu-18.04 + strategy: + matrix: + target: [proxy, controller, web, cni-plugin, debug, cli-bin, grafana] + name: Docker build (${{ matrix.target }}) steps: - name: Checkout code # actions/checkout@v2 @@ -24,39 +28,35 @@ jobs: . bin/_docker.sh echo ::set-env name=DOCKER_REGISTRY::$DOCKER_REGISTRY - - name: Setup SSH config for Packet - if: github.event_name == 'push' || !github.event.pull_request.head.repo.fork - run: | - mkdir -p ~/.ssh/ - touch ~/.ssh/id && chmod 600 ~/.ssh/id - echo "${{ secrets.DOCKER_SSH_CONFIG }}" > ~/.ssh/config - echo "${{ secrets.DOCKER_PRIVATE_KEY }}" > ~/.ssh/id - echo "${{ secrets.DOCKER_KNOWN_HOSTS }}" > ~/.ssh/known_hosts - ssh linkerd-docker docker version - echo ::set-env name=DOCKER_HOST::ssh://linkerd-docker + echo ::set-env name=DOCKER_BUILDKIT_CACHE::${{ runner.temp }}/.buildx-cache + - name: Cache docker layers + # actions/cache@v2.0.0 + uses: actions/cache@b820478 + with: + path: ${{ env.DOCKER_BUILDKIT_CACHE }} + key: ${{ runner.os }}-buildx-${{ matrix.target }}-${{ env.TAG }} + restore-keys: | + ${{ runner.os }}-buildx-${{ matrix.target }}- - name: Build docker images env: DOCKER_TRACE: 1 run: | - export PATH="`pwd`/bin:$PATH" - bin/docker-build + docker buildx create --driver docker-container --use + bin/docker-build-${{ matrix.target }} + - name: Prune docker layers cache + # changes generate new images while the existing ones don't get removed + # so we manually do that to avoid bloating the cache + run: bin/docker-cache-prune - name: Create artifact with CLI and image archives env: ARCHIVES: /home/runner/archives run: | mkdir -p $ARCHIVES - - for image in proxy controller web cni-plugin debug cli-bin grafana; do - docker save "$DOCKER_REGISTRY/$image:$TAG" > $ARCHIVES/$image.tar || tee save_fail & - done - + docker save "$DOCKER_REGISTRY/${{ matrix.target }}:$TAG" > $ARCHIVES/${{ matrix.target }}.tar # save windows cli into artifacts - cp -r ./target/cli/windows/linkerd $ARCHIVES/linkerd-windows.exe - - # Wait for `docker save` background processes to complete. Exit early - # if any job failed. - wait < <(jobs -p) - test -f save_fail && exit 1 || true + if [ '${{ matrix.target }}' == 'cli-bin' ]; then + cp -r ./target/cli/windows/linkerd $ARCHIVES/linkerd-windows.exe + fi # `with.path` values do not support environment variables yet, so an # absolute path is used here. # @@ -92,7 +92,6 @@ jobs: - name: Run CLI Integration tests run: | go test --failfast --mod=readonly ".\test\cli" --linkerd=$PWD\image-archives\linkerd-windows.exe --cli-tests -v - # todo: Keep in sync with `release.yml` kind_integration_tests: strategy: matrix: @@ -127,31 +126,12 @@ jobs: . bin/_docker.sh echo ::set-env name=DOCKER_REGISTRY::$DOCKER_REGISTRY - - name: Setup SSH config for Packet - if: github.event_name == 'push' || !github.event.pull_request.head.repo.fork - run: | - mkdir -p ~/.ssh/ - touch ~/.ssh/id && chmod 600 ~/.ssh/id - echo "${{ secrets.DOCKER_SSH_CONFIG }}" > ~/.ssh/config - echo "${{ secrets.DOCKER_PRIVATE_KEY }}" > ~/.ssh/id - echo "${{ secrets.DOCKER_KNOWN_HOSTS }}" > ~/.ssh/known_hosts - - name: Download image archives (Forked repositories) - if: github.event_name == 'pull_request' && github.event.pull_request.head.repo.fork + - name: Download image archives # actions/download-artifact@v1 uses: actions/download-artifact@18f0f59 with: name: image-archives - name: Load cli-bin image into local docker images - if: github.event_name == 'push' || !github.event.pull_request.head.repo.fork - run: | - # `docker load` only accepts input from STDIN, so pipe the image - # archive into the command. - # - # In order to pipe the image archive, set `DOCKER_HOST` for a single - # command and `docker save` the CLI image from the Packet host. - DOCKER_HOST=ssh://linkerd-docker docker save "$DOCKER_REGISTRY/cli-bin:$TAG" | docker load - - name: Load cli-bin image into local docker images (Forked repositories) - if: github.event_name == 'pull_request' && github.event.pull_request.head.repo.fork run: docker load < image-archives/cli-bin.tar - name: Install CLI run: | @@ -162,10 +142,5 @@ jobs: # Validate the CLI version matches the current build tag. [[ "$TAG" == "$($HOME/.linkerd version --short --client)" ]] - name: Run integration tests - if: github.event_name == 'push' || !github.event.pull_request.head.repo.fork - run: | - bin/tests --images --images-host ssh://linkerd-docker --name ${{ matrix.integration_test }} "$HOME/.linkerd" - - name: Run integration tests (Forked repositories) - if: github.event_name == 'pull_request' && github.event.pull_request.head.repo.fork run: | bin/tests --images --name ${{ matrix.integration_test }} "$HOME/.linkerd" diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 9705035a7..5f72f5a08 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -1,10 +1,6 @@ # # This file is a mostly a concatenation of `kind_integration.yml` and -# `cloud_integration.yml`, specifically for release. Once GitHub Actions -# supports YAML anchors, we should be able to share most of the content -# between these files: -# https://github.community/t5/GitHub-Actions/Support-for-YAML-anchors/m-p/30336 -# +# `cloud_integration.yml`, specifically for release. name: Release on: @@ -13,53 +9,14 @@ on: - "*" env: GH_ANNOTATION: true + DOCKER_BUILDKIT: 1 jobs: - # todo: Keep in sync with `cloud_integration.yml` docker_build: - name: Docker build runs-on: ubuntu-18.04 - steps: - - name: Checkout code - # actions/checkout@v2 - uses: actions/checkout@722adc6 - - name: Setup SSH config for Packet - run: | - mkdir -p ~/.ssh/ - touch ~/.ssh/id && chmod 600 ~/.ssh/id - echo "${{ secrets.DOCKER_SSH_CONFIG }}" > ~/.ssh/config - echo "${{ secrets.DOCKER_PRIVATE_KEY }}" > ~/.ssh/id - echo "${{ secrets.DOCKER_KNOWN_HOSTS }}" > ~/.ssh/known_hosts - ssh linkerd-docker docker version - - name: Build docker images - env: - DOCKER_HOST: ssh://linkerd-docker - DOCKER_TRACE: 1 - run: | - export PATH="`pwd`/bin:$PATH" - bin/docker-build - - name: Create artifact with Windows CLI binary - env: - ARCHIVES: /home/runner/archives - run: | - mkdir -p $ARCHIVES - - # save windows cli into artifacts - cp -r ./target/cli/windows/linkerd $ARCHIVES/linkerd-windows.exe - # `with.path` values do not support environment variables yet, so an - # absolute path is used here. - # - # https://github.com/actions/upload-artifact/issues/8 - - name: Upload artifact - # actions/upload-artifact@v1 - uses: actions/upload-artifact@3446296 - with: - name: image-archives - path: /home/runner/archives - # todo: Keep in sync with `cloud_integration.yml` - docker_push: - name: Docker push - runs-on: ubuntu-18.04 - needs: [docker_build] + strategy: + matrix: + target: [proxy, controller, web, cni-plugin, debug, cli-bin, grafana] + name: Docker build (${{ matrix.target }}) steps: - name: Checkout code # actions/checkout@v2 @@ -68,6 +25,48 @@ jobs: run: | . bin/_tag.sh echo ::set-env name=TAG::$(CI_FORCE_CLEAN=1 bin/root-tag) + + . bin/_docker.sh + echo ::set-env name=DOCKER_REGISTRY::$DOCKER_REGISTRY + echo ::set-env name=DOCKER_BUILDKIT_CACHE::${{ runner.temp }}/.buildx-cache + - name: Cache docker layers + # actions/cache@v2.0.0 + uses: actions/cache@b820478 + with: + path: ${{ env.DOCKER_BUILDKIT_CACHE }} + key: ${{ runner.os }}-buildx-${{ matrix.target }}-${{ env.TAG }} + restore-keys: | + ${{ runner.os }}-buildx-${{ matrix.target }}- + - name: Build docker images + env: + DOCKER_TRACE: 1 + run: | + docker buildx create --driver docker-container --use + bin/docker-build-${{ matrix.target }} + - name: Prune docker layers cache + # changes generate new images while the existing ones don't get removed + # so we manually do that to avoid bloating the cache + run: bin/docker-cache-prune + - name: Create artifact with CLI + # windows_static_cli_tests below needs this because it can't create linux containers + # inside windows + if: matrix.target == 'cli-bin' + env: + ARCHIVES: /home/runner/archives + run: | + mkdir -p $ARCHIVES + cp -r ./target/cli/windows/linkerd $ARCHIVES/linkerd-windows.exe + # `with.path` values do not support environment variables yet, so an + # absolute path is used here. + # + # https://github.com/actions/upload-artifact/issues/8 + - name: Upload artifact + if: matrix.target == 'cli-bin' + # actions/upload-artifact@v1 + uses: actions/upload-artifact@3446296 + with: + name: image-archives + path: /home/runner/archives - name: Configure gcloud # linkerd/linkerd2-action-gcloud@v1.0.1 uses: linkerd/linkerd2-action-gcloud@308c4df @@ -75,23 +74,12 @@ jobs: cloud_sdk_service_account_key: ${{ secrets.CLOUD_SDK_SERVICE_ACCOUNT_KEY }} gcp_project: ${{ secrets.GCP_PROJECT }} gcp_zone: ${{ secrets.GCP_ZONE }} - - name: Docker SSH setup - run: | - mkdir -p ~/.ssh/ - touch ~/.ssh/id && chmod 600 ~/.ssh/id - echo "${{ secrets.DOCKER_SSH_CONFIG }}" > ~/.ssh/config - echo "${{ secrets.DOCKER_PRIVATE_KEY }}" > ~/.ssh/id - echo "${{ secrets.DOCKER_KNOWN_HOSTS }}" > ~/.ssh/known_hosts - ssh linkerd-docker docker version - name: Push docker images to registry - env: - DOCKER_HOST: ssh://linkerd-docker run: | - export PATH="`pwd`/bin:$PATH" - bin/docker-push-deps - bin/docker-push $TAG - bin/docker-retag-all $TAG main - bin/docker-push main + . bin/_docker.sh + docker_push "${{ matrix.target }}" "$TAG" + docker_retag "${{ matrix.target }}" "$TAG" main + docker_push "${{ matrix.target }}" main # todo: Keep in sync with `kind_integration.yml` windows_static_cli_tests: name: Static CLI tests (windows) @@ -117,7 +105,6 @@ jobs: - name: Run CLI Integration tests run: | go test --failfast --mod=readonly ".\test\cli" --linkerd=$PWD\image-archives\linkerd-windows.exe --cli-tests -v - # todo: Keep in sync with `kind_integration.yml` kind_integration_tests: strategy: matrix: @@ -147,41 +134,21 @@ jobs: ${{ runner.os }}-go- - name: Set environment variables from scripts run: | - . bin/_tag.sh - echo ::set-env name=TAG::$(CI_FORCE_CLEAN=1 bin/root-tag) - - . bin/_docker.sh - echo ::set-env name=DOCKER_REGISTRY::$DOCKER_REGISTRY - - name: Setup SSH config for Packet - run: | - mkdir -p ~/.ssh/ - touch ~/.ssh/id && chmod 600 ~/.ssh/id - echo "${{ secrets.DOCKER_SSH_CONFIG }}" > ~/.ssh/config - echo "${{ secrets.DOCKER_PRIVATE_KEY }}" > ~/.ssh/id - echo "${{ secrets.DOCKER_KNOWN_HOSTS }}" > ~/.ssh/known_hosts - - name: Load cli-bin image into local docker images - run: | - # `docker load` only accepts input from STDIN, so pipe the image - # archive into the command. - # - # In order to pipe the image archive, set `DOCKER_HOST` for a single - # command and `docker save` the CLI image from the Packet host. - DOCKER_HOST=ssh://linkerd-docker docker save "$DOCKER_REGISTRY/cli-bin:$TAG" | docker load - - name: Install CLI - run: | - # Copy the CLI out of the local cli-bin container. - container_id=$(docker create "$DOCKER_REGISTRY/cli-bin:$TAG") - docker cp $container_id:/out/linkerd-linux $HOME/.linkerd - - # Validate the CLI version matches the current build tag. - [[ "$TAG" == "$($HOME/.linkerd version --short --client)" ]] + TAG="$(CI_FORCE_CLEAN=1 bin/root-tag)" + CMD="$PWD/target/release/linkerd2-cli-$TAG-linux" + echo "::set-env name=CMD::$CMD" + echo "::set-env name=TAG::$TAG" - name: Run integration tests - run: bin/tests --images --images-host ssh://linkerd-docker --name ${{ matrix.integration_test }} "$HOME/.linkerd" + run: | + bin/docker-pull-binaries $TAG + # Validate the CLI version matches the current build tag. + [[ "$TAG" == "$($CMD version --short --client)" ]] + bin/tests --images --name ${{ matrix.integration_test }} "$CMD" # todo: Keep in sync with `cloud_integration.yml` cloud_integration_tests: name: Cloud integration tests runs-on: ubuntu-18.04 - needs: [docker_push] + needs: [docker_build] steps: - name: Checkout code # actions/checkout@v2 @@ -198,13 +165,13 @@ jobs: id: install_cli run: | TAG="$(CI_FORCE_CLEAN=1 bin/root-tag)" - image="gcr.io/linkerd-io/cli-bin:$TAG" - id=$(bin/docker create $image) - bin/docker cp "$id:/out/linkerd-linux" "$HOME/.linkerd" - $HOME/.linkerd version --client + CMD="$PWD/target/release/linkerd2-cli-$TAG-linux" + bin/docker-pull-binaries $TAG + $CMD version --client # validate CLI version matches the repo - [[ "$TAG" == "$($HOME/.linkerd version --short --client)" ]] + [[ "$TAG" == "$($CMD version --short --client)" ]] echo "Installed Linkerd CLI version: $TAG" + echo "::set-env name=CMD::$CMD" echo "::set-output name=tag::$TAG" - name: Create GKE cluster # linkerd/linkerd2-action-gcloud@v1.0.1 @@ -221,13 +188,11 @@ jobs: env: GITCOOKIE_SH: ${{ secrets.GITCOOKIE_SH }} run: | - export PATH="`pwd`/bin:$PATH" echo "$GITCOOKIE_SH" | bash - version="$($HOME/.linkerd version --client --short | tr -cd '[:alnum:]-')" - bin/tests --skip-kind-create "$HOME/.linkerd" + bin/tests --skip-kind-create "$CMD" - name: CNI tests run: | - export TAG="$($HOME/.linkerd version --client --short)" + export TAG="$($CMD version --client --short)" go test -cover -race -v -mod=readonly ./cni-plugin/test -integration-tests choco_pack: # only runs for stable tags. The conditionals are at each step level instead of the job level diff --git a/.github/workflows/static_checks.yml b/.github/workflows/static_checks.yml index 853f19faf..6ce62baaf 100644 --- a/.github/workflows/static_checks.yml +++ b/.github/workflows/static_checks.yml @@ -8,29 +8,6 @@ on: branches: - main jobs: - go_dependencies: - name: Go dependencies - runs-on: ubuntu-18.04 - steps: - - name: Checkout code - # actions/checkout@v2 - uses: actions/checkout@722adc6 - - name: Dump env - run: env | sort - - name: Dump GitHub context - env: - GITHUB_CONTEXT: ${{ toJson(github) }} - run: echo "$GITHUB_CONTEXT" - - name: Dump job context - env: - JOB_CONTEXT: ${{ toJson(job) }} - run: echo "$JOB_CONTEXT" - - name: Validate go deps - run: | - . bin/_tag.sh - for f in $( grep -lR --include=Dockerfile\* go-deps: . ) ; do - validate_go_deps_tag $f - done go_lint: name: Go lint runs-on: ubuntu-18.04 diff --git a/BUILD.md b/BUILD.md index 887150b8c..5d06c8797 100644 --- a/BUILD.md +++ b/BUILD.md @@ -19,7 +19,6 @@ about testing from source can be found in the [TEST.md](TEST.md) guide. - [Rust](#rust) - [Dependencies](#dependencies) - [Updating protobuf dependencies](#updating-protobuf-dependencies) - - [Updating Docker dependencies](#updating-docker-dependencies) - [Updating ServiceProfile generated code](#updating-serviceprofile-generated-code) - [Helm Chart](#helm-chart) @@ -365,22 +364,6 @@ DOCKER_TRACE=1 bin/docker-build-proxy bin/protoc-go.sh ``` -### Updating Docker dependencies - -The Go Docker images rely on base dependency images with hard-coded SHA's: - -`gcr.io/linkerd-io/go-deps` depends on - -- [`go.mod`](go.mod) -- [`Dockerfile-go-deps`](Dockerfile-go-deps) - -When Go dependencies change, run the following: - -```bash -go mod tidy -bin/update-go-deps-shas -``` - ### Updating ServiceProfile generated code The [ServiceProfile client code](./controller/gen/client) is generated by @@ -541,12 +524,6 @@ build_architecture "workflow.yml" -> "docker-retag-all"; "workflow.yml" -> "lint"; - "update-go-deps-shas" -> "_tag.sh"; - "update-go-deps-shas" -> "cli/Dockerfile-bin"; - "update-go-deps-shas" -> "controller/Dockerfile"; - "update-go-deps-shas" -> "grafana/Dockerfile"; - "update-go-deps-shas" -> "web/Dockerfile"; - "web" -> "go-run"; } build_architecture diff --git a/Dockerfile-proxy b/Dockerfile-proxy index c450fb121..5c3531461 100644 --- a/Dockerfile-proxy +++ b/Dockerfile-proxy @@ -1,5 +1,13 @@ ARG RUNTIME_IMAGE=debian:buster-20200514-slim +# Precompile key slow-to-build dependencies +FROM golang:1.14.2-alpine as go-deps +WORKDIR /linkerd-build +COPY go.mod go.sum ./ +COPY bin/install-deps bin/ +RUN go mod download +RUN ./bin/install-deps + FROM debian:buster-20200514-slim as fetch RUN apt-get update && apt-get install -y ca-certificates curl jq WORKDIR /build @@ -9,7 +17,7 @@ RUN (proxy=$(bin/fetch-proxy $(cat proxy-version)) && \ mv "$proxy" linkerd2-proxy) ## compile proxy-identity agent -FROM gcr.io/linkerd-io/go-deps:61149d15 as golang +FROM go-deps as golang WORKDIR /linkerd-build COPY pkg/flags pkg/flags COPY pkg/tls pkg/tls diff --git a/bin/_docker.sh b/bin/_docker.sh index 9a2022c4f..4bd89513a 100644 --- a/bin/_docker.sh +++ b/bin/_docker.sh @@ -14,6 +14,12 @@ export DOCKER_REGISTRY=${DOCKER_REGISTRY:-gcr.io/linkerd-io} # When set, causes docker's build output to be emitted to stderr. export DOCKER_TRACE=${DOCKER_TRACE:-} +# When set, use `docker buildx` and use the github actions cache to store/retrieve images +export DOCKER_BUILDKIT=${DOCKER_BUILDKIT:-} + +# buildx cache directory. Needed if DOCKER_BUILDKIT is used +export DOCKER_BUILDKIT_CACHE=${DOCKER_BUILDKIT_CACHE:-} + docker_repo() { repo=$1 @@ -42,12 +48,27 @@ docker_build() { rootdir=$( cd "$bindir"/.. && pwd ) - log_debug " :; docker build $rootdir -t $repo:$tag -f $file $*" - docker build "$rootdir" \ - -t "$repo:$tag" \ - -f "$file" \ - "$@" \ - > "$output" + if [ -n "$DOCKER_BUILDKIT" ]; then + cache_params="" + if [ -n "$DOCKER_BUILDKIT_CACHE" ]; then + cache_params="--cache-from type=local,src=${DOCKER_BUILDKIT_CACHE} --cache-to type=local,dest=${DOCKER_BUILDKIT_CACHE},mode=max" + fi + log_debug " :; docker buildx $rootdir $cache_params --load -t $repo:$tag -f $file $*" + # shellcheck disable=SC2086 + docker buildx build "$rootdir" $cache_params \ + --load \ + -t "$repo:$tag" \ + -f "$file" \ + "$@" \ + > "$output" + else + log_debug " :; docker build $rootdir -t $repo:$tag -f $file $*" + docker build "$rootdir" \ + -t "$repo:$tag" \ + -f "$file" \ + "$@" \ + > "$output" + fi echo "$repo:$tag" } diff --git a/bin/_tag.sh b/bin/_tag.sh index 32bdea2c8..c843ded79 100644 --- a/bin/_tag.sh +++ b/bin/_tag.sh @@ -6,12 +6,6 @@ git_sha_head() { git rev-parse --short=8 HEAD } -go_deps_sha() { - bindir=$( cd "${BASH_SOURCE[0]%/*}" && pwd ) - rootdir=$( cd "$bindir"/.. && pwd ) - cat "$rootdir/go.mod" "$rootdir/Dockerfile-go-deps" | shasum - | awk '{print $1}' |cut -c 1-8 -} - clean_head() { [ -n "${CI_FORCE_CLEAN:-}" ] || git diff-index --quiet HEAD -- } @@ -43,33 +37,3 @@ clean_head_root_tag() { exit 3 fi } - -validate_tag() { - file=$1 - shift - - image=$1 - shift - - sha=$1 - shift - - dockerfile_tag=$(grep -oe "$image"':[^ ]*' "$file") || true - deps_tag="$image:$sha" - if [ -n "$dockerfile_tag" ] && [ "$dockerfile_tag" != "$deps_tag" ]; then - echo "Tag in \"$file\" does not match source tree: -$dockerfile_tag (\"$file\") -$deps_tag (source)" - return 3 - fi -} - -# This function should be called by any docker-build-* script that relies on Go -# dependencies. To confirm the set of scripts that should call this function, -# run: -# $ grep -ER 'docker-build-go-deps' . - -validate_go_deps_tag() { - file=$1 - validate_tag "$file" gcr.io/linkerd-io/go-deps "$(go_deps_sha)" -} diff --git a/bin/docker-build-cli-bin b/bin/docker-build-cli-bin index 843251d75..01377e314 100755 --- a/bin/docker-build-cli-bin +++ b/bin/docker-build-cli-bin @@ -16,13 +16,6 @@ rootdir=$( cd "$bindir"/.. && pwd ) . "$bindir"/_tag.sh dockerfile=$rootdir/cli/Dockerfile-bin - -validate_go_deps_tag "$dockerfile" - -( - "$bindir"/docker-build-go-deps -) >/dev/null - tag=$(head_root_tag) docker_build cli-bin "$tag" "$dockerfile" --build-arg LINKERD_VERSION="$tag" IMG=$(docker_repo cli-bin):$tag diff --git a/bin/docker-build-cni-plugin b/bin/docker-build-cni-plugin index 25a9622f5..5674a9d9b 100755 --- a/bin/docker-build-cni-plugin +++ b/bin/docker-build-cni-plugin @@ -17,11 +17,8 @@ rootdir=$( cd "$bindir"/.. && pwd ) dockerfile=$rootdir/cni-plugin/Dockerfile -validate_go_deps_tag "$dockerfile" - ( "$bindir"/docker-build-base - "$bindir"/docker-build-go-deps ) >/dev/null docker_build cni-plugin "$(head_root_tag)" "$dockerfile" diff --git a/bin/docker-build-controller b/bin/docker-build-controller index f12af8fe8..eb41fa0d2 100755 --- a/bin/docker-build-controller +++ b/bin/docker-build-controller @@ -16,12 +16,5 @@ rootdir=$( cd "$bindir"/.. && pwd ) . "$bindir"/_tag.sh dockerfile=$rootdir/controller/Dockerfile - -validate_go_deps_tag "$dockerfile" - -( - "$bindir"/docker-build-go-deps -) >/dev/null - tag=$(head_root_tag) docker_build controller "$tag" "$dockerfile" --build-arg LINKERD_VERSION="$tag" diff --git a/bin/docker-build-go-deps b/bin/docker-build-go-deps deleted file mode 100755 index 4b4e63056..000000000 --- a/bin/docker-build-go-deps +++ /dev/null @@ -1,26 +0,0 @@ -#!/usr/bin/env bash - -# Builds (or pulls) our go-deps docker image. - -set -eu - -if [ $# -ne 0 ]; then - echo "no arguments allowed for ${0##*/}, given: $*" >&2 - exit 64 -fi - -bindir=$( cd "${BASH_SOURCE[0]%/*}" && pwd ) - -# shellcheck source=_docker.sh -. "$bindir"/_docker.sh -# shellcheck source=_tag.sh -. "$bindir"/_tag.sh - -tag=$(go_deps_sha) - -if (docker_pull go-deps "$tag"); then - echo "$(docker_repo go-deps):$tag" -else - rootdir=$( cd "$bindir"/.. && pwd ) - docker_build go-deps "$tag" "$rootdir/Dockerfile-go-deps" -fi diff --git a/bin/docker-build-proxy b/bin/docker-build-proxy index 2e3f61796..ff248fbc3 100755 --- a/bin/docker-build-proxy +++ b/bin/docker-build-proxy @@ -17,12 +17,6 @@ rootdir=$( cd "$bindir"/.. && pwd ) dockerfile=$rootdir/Dockerfile-proxy -validate_go_deps_tag "$dockerfile" - -( - "$bindir"/docker-build-go-deps -) >/dev/null - get_extra_options() { options= for var in http_proxy https_proxy no_proxy; do diff --git a/bin/docker-build-web b/bin/docker-build-web index 0b63408cb..55ad1d650 100755 --- a/bin/docker-build-web +++ b/bin/docker-build-web @@ -16,12 +16,5 @@ rootdir=$( cd "$bindir"/.. && pwd ) . "$bindir"/_tag.sh dockerfile=$rootdir/web/Dockerfile - -validate_go_deps_tag "$dockerfile" - -( - "$bindir"/docker-build-go-deps -) >/dev/null - tag=$(head_root_tag) docker_build web "$tag" "$dockerfile" --build-arg LINKERD_VERSION="$tag" diff --git a/bin/docker-cache-prune b/bin/docker-cache-prune new file mode 100755 index 000000000..75821ade8 --- /dev/null +++ b/bin/docker-cache-prune @@ -0,0 +1,21 @@ +#!/usr/bin/env bash + +# This script deletes all the files under the buildkit cache blob directory that +# are not referred to in the cache manifest file. + +set -eu + +export DOCKER_BUILDKIT_CACHE=${DOCKER_BUILDKIT_CACHE:-} + +manifest_sha=$(jq -r .manifests[0].digest < "${DOCKER_BUILDKIT_CACHE}/index.json") +manifest=${manifest_sha#"sha256:"} +files=("$manifest") +while IFS= read -r line; do files+=("$line"); done <<< "$(jq -r '.manifests[].digest | sub("^sha256:"; "")' < "${DOCKER_BUILDKIT_CACHE}/blobs/sha256/$manifest")" +for file in "${DOCKER_BUILDKIT_CACHE}"/blobs/sha256/*; do + name=$(basename "$file") + # shellcheck disable=SC2199 + if [[ ! "${files[@]}" =~ ${name} ]]; then + printf 'pruned from cache: %s\n' "$file" + rm -f "$file" + fi +done diff --git a/bin/docker-images b/bin/docker-images index c1866947f..3ba062e49 100755 --- a/bin/docker-images +++ b/bin/docker-images @@ -21,5 +21,3 @@ tag=$(head_root_tag) for img in cli-bin cni-plugin controller debug grafana proxy web ; do docker_image $img "$tag" done - -docker_image go-deps "$(go_deps_sha)" diff --git a/bin/docker-pull-deps b/bin/docker-pull-base similarity index 64% rename from bin/docker-pull-deps rename to bin/docker-pull-base index 958b25c39..546664350 100755 --- a/bin/docker-pull-deps +++ b/bin/docker-pull-base @@ -6,8 +6,5 @@ bindir=$( cd "${BASH_SOURCE[0]%/*}" && pwd ) # shellcheck source=_docker.sh . "$bindir"/_docker.sh -# shellcheck source=_tag.sh -. "$bindir"/_tag.sh docker_pull base 2020-06-08.01 || true -docker_pull go-deps "$(go_deps_sha)" || true diff --git a/bin/docker-push-deps b/bin/docker-push-base similarity index 65% rename from bin/docker-push-deps rename to bin/docker-push-base index 5e92cfc64..d043b9ddf 100755 --- a/bin/docker-push-deps +++ b/bin/docker-push-base @@ -6,8 +6,5 @@ bindir=$( cd "${BASH_SOURCE[0]%/*}" && pwd ) # shellcheck source=_docker.sh . "$bindir"/_docker.sh -# shellcheck source=_tag.sh -. "$bindir"/_tag.sh docker_push base 2020-06-08.01 -docker_push go-deps "$(go_deps_sha)" diff --git a/Dockerfile-go-deps b/bin/install-deps old mode 100644 new mode 100755 similarity index 80% rename from Dockerfile-go-deps rename to bin/install-deps index 528f79f64..a50a41000 --- a/Dockerfile-go-deps +++ b/bin/install-deps @@ -1,25 +1,15 @@ -# Go dependencies -# -# Fetches all required Go dependencies. All Linkerd sources are omitted from the -# resulting image so that artifacts may be built from source over this image. -# -# When this file is changed, run `bin/update-go-deps-shas`. +#!/usr/bin/env sh -FROM golang:1.14.2 +# This script is used in the multiple Dockerfiles for caching +# some of the slow-to-build go dependencies. -WORKDIR /linkerd-build +set -eu -COPY go.mod go.sum ./ +bindir=$( cd "${0%/*}" && pwd ) +rootdir=$( cd "$bindir"/.. && pwd ) +cd "$rootdir" -RUN go mod download - -# Precompile key slow-to-build dependencies. This list doesn't need to be -# complete for the build to work correctly; the completeness of this list -# only affects the speed of incremental rebuilds of Dependent Dockerfiles. -# -# This list was derived from the output of `find /go/pkg -type f` -# after building the controller. -RUN CGO_ENABLED=0 GOOS=linux go install -mod=readonly \ +CGO_ENABLED=0 GOOS=linux go install -mod=readonly \ github.com/golang/protobuf/jsonpb \ github.com/grpc-ecosystem/go-grpc-prometheus \ github.com/prometheus/client_golang/api \ diff --git a/bin/update-go-deps-shas b/bin/update-go-deps-shas deleted file mode 100755 index 0279dead7..000000000 --- a/bin/update-go-deps-shas +++ /dev/null @@ -1,13 +0,0 @@ -#!/usr/bin/env bash - -set -eu - -# Updates the tag for `linkerd-io/go-deps` across all Dockerfiles in this repository. - -# shellcheck source=_tag.sh -sha=$(. "${0%/*}"/_tag.sh ; go_deps_sha) - -for f in $( grep -lR --include=Dockerfile\* go-deps: "$(cd "${0%/*}"/.. && pwd)" | xargs ) ; do - sed -E -i.bak -e "s|linkerd-io/go-deps:[^ ]+|linkerd-io/go-deps:$sha|" "$f" - rm "$f".bak -done diff --git a/cli/Dockerfile-bin b/cli/Dockerfile-bin index 9fc53ae2e..c2a3a3791 100644 --- a/cli/Dockerfile-bin +++ b/cli/Dockerfile-bin @@ -1,5 +1,13 @@ +# Precompile key slow-to-build dependencies +FROM golang:1.14.2-alpine as go-deps +WORKDIR /linkerd-build +COPY go.mod go.sum ./ +COPY bin/install-deps bin/ +RUN go mod download +RUN ./bin/install-deps + ## compile binaries -FROM gcr.io/linkerd-io/go-deps:61149d15 as golang +FROM go-deps as golang WORKDIR /linkerd-build COPY cli cli COPY charts charts diff --git a/cni-plugin/Dockerfile b/cni-plugin/Dockerfile index 37e39faff..6d0e9043c 100644 --- a/cni-plugin/Dockerfile +++ b/cni-plugin/Dockerfile @@ -1,5 +1,13 @@ +# Precompile key slow-to-build dependencies +FROM golang:1.14.2-alpine as go-deps +WORKDIR /linkerd-build +COPY go.mod go.sum ./ +COPY bin/install-deps bin/ +RUN go mod download +RUN ./bin/install-deps + ## compile cni-plugin utility -FROM gcr.io/linkerd-io/go-deps:61149d15 as golang +FROM go-deps as golang WORKDIR /linkerd-build COPY pkg pkg COPY controller controller diff --git a/controller/Dockerfile b/controller/Dockerfile index 3781e92b6..dd2b70d74 100644 --- a/controller/Dockerfile +++ b/controller/Dockerfile @@ -1,5 +1,13 @@ +# Precompile key slow-to-build dependencies +FROM golang:1.14.2-alpine as go-deps +WORKDIR /linkerd-build +COPY go.mod go.sum ./ +COPY bin/install-deps bin/ +RUN go mod download +RUN ./bin/install-deps + ## compile controller service -FROM gcr.io/linkerd-io/go-deps:61149d15 as golang +FROM go-deps as golang WORKDIR /linkerd-build COPY controller/gen controller/gen COPY pkg pkg diff --git a/web/Dockerfile b/web/Dockerfile index 7fc12f16a..9b5a4eedc 100644 --- a/web/Dockerfile +++ b/web/Dockerfile @@ -1,3 +1,11 @@ +# Precompile key slow-to-build dependencies +FROM golang:1.14.2-alpine as go-deps +WORKDIR /linkerd-build +COPY go.mod go.sum ./ +COPY bin/install-deps bin/ +RUN go mod download +RUN ./bin/install-deps + ## bundle web assets FROM node:14-buster as webpack-bundle RUN curl -o- -L https://yarnpkg.com/install.sh | bash -s -- --version 1.21.1 --network-concurrency 1 @@ -21,7 +29,7 @@ COPY web/app ./web/app RUN ./bin/web build ## compile go server -FROM gcr.io/linkerd-io/go-deps:61149d15 as golang +FROM go-deps as golang WORKDIR /linkerd-build RUN mkdir -p web COPY web/main.go web