Run integration tests for forked repos (#4002)

* Allow CI to run concurrent builds in master

Fixes #3911

Refactors the `cloud_integration` test to run in separate GKE clusters
that are created and torn down on the fly.
It leverages a new "gcloud" github action that is also used to set up
gcloud in other build steps (`docker_deploy` and `chart_deploy`).

The action also generates unique names for those clusters, based on the
git commit SHA and `run_id`, a recently introduced variable that is
unique per CI run and available to all the jobs.
This fixes part of #3635 in that CI runs on the same SHA don't interfere
with one another (in the `cloud_integration` test; still to do for
`kind_integration`).

The "gcloud" GH action is supported by `.github/actions/gcloud/index.js`
that has a couple of dependencies. To avoid having to commit
`node_modules`, after every change to that file one must run
```bash
# only needed the first time
npm i -g @zeit/ncc

cd .github/actions/gcloud
ncc build index.js
```
which generates the self-contained file
`.github/actions/gcloud/dist/index.js`.
(This last part might get easier in the future after other refactorings
outside this PR).

* Run integration tests for forked repos

Signed-off-by: Kevin Leimkuhler <kevin@kleimkuhler.com>

* Address reviews

Signed-off-by: Kevin Leimkuhler <kevin@kleimkuhler.com>

* Address more reviews

Signed-off-by: Kevin Leimkuhler <kevin@kleimkuhler.com>

* Move some conditionals to jobs

Signed-off-by: Kevin Leimkuhler <kevin@kleimkuhler.com>

* Change job name

Signed-off-by: Kevin Leimkuhler <kevin@kleimkuhler.com>

* Move more conditionals to job level

Signed-off-by: Kevin Leimkuhler <kevin@kleimkuhler.com>

* Added more flags to 'gcloud container clusters create' and consolidated
'create' and 'destroy' into ' action'

* Run kind cleanup only for non-forked PRs

Signed-off-by: Kevin Leimkuhler <kevin@kleimkuhler.com>

* Got rid of cloud_cleanup by using a post hook in the gcloud action

* Removed cluster naming responsibility from the gcloud action

* Consolidate .gitignore statements

* Removed bin/_gcp.sh

* Change name of Kind int. test job

Signed-off-by: Kevin Leimkuhler <kevin@kleimkuhler.com>

* Ensure `kind_cleanup` still runs on cancelled host CI runs

Signed-off-by: Kevin Leimkuhler <kevin@kleimkuhler.com>

* Add reviews

Signed-off-by: Kevin Leimkuhler <kevin@kleimkuhler.com>

* Update workflow comment

Signed-off-by: Kevin Leimkuhler <kevin@kleimkuhler.com>

* Split index.js into setup.js and destroy.js

* trigger build

* Moved the gcloud action into its own repo

* Full version for the gcloud GH action

* Rebase back to master

Signed-off-by: Kevin Leimkuhler <kevin@kleimkuhler.com>

* Remvoe additional changes

Signed-off-by: Kevin Leimkuhler <kevin@kleimkuhler.com>

* Remove additional changes

Signed-off-by: Kevin Leimkuhler <kevin@kleimkuhler.com>

* Trigger CI

Signed-off-by: Kevin Leimkuhler <kevin@kleimkuhler.com>

Co-authored-by: Alejandro Pedraza <alejandro.pedraza@gmail.com>

.github/workflows/workflow.yml

@@ -17,14 +17,19 @@ on:
 # go_lint
 # js_unit_tests
 
-# Docker build and integration tests for every master/tag push and linkerd org PR
+# master/tag push and linkerd org PR: Docker build and integration tests
 #
 # docker_pull
 # docker_build
 # kind_setup
-#   -> kind_integration
+#   -> kind_integration_host
 #     -> kind_cleanup
 
+# Forked repository PR: Docker build and integration tests
+#
+# docker_build
+#   -> kind_integration_github
+
 # Docker deploy and cloud integration tests for every master/tag push
 #
 #       -> docker_deploy
@@ -135,18 +140,17 @@ jobs:
   # Docker build and kind integration tests run for:
   # - every master push
   # - every tag push
-  # - every PR from a linkerd org member
+  # - every PR
   #
 
   docker_pull:
     name: Docker pull
+    if: github.event_name != 'pull_request' || !github.event.pull_request.head.repo.fork
     runs-on: ubuntu-18.04
     steps:
     - name: Checkout code
-      if: github.event_name != 'pull_request' || !github.event.pull_request.head.repo.fork
       uses: actions/checkout@v2
     - name: Docker SSH setup
-      if: github.event_name != 'pull_request' || !github.event.pull_request.head.repo.fork
       run: |
         mkdir -p ~/.ssh/
         touch ~/.ssh/id && chmod 600 ~/.ssh/id
@@ -155,7 +159,6 @@ jobs:
         echo "${{ secrets.DOCKER_KNOWN_HOSTS }}" > ~/.ssh/known_hosts
         ssh linkerd-docker docker version
     - name: Docker pull
-      if: github.event_name != 'pull_request' || !github.event.pull_request.head.repo.fork
       env:
         DOCKER_HOST: ssh://linkerd-docker
       run: |
@@ -165,9 +168,10 @@ jobs:
   docker_build:
     name: Docker build
     runs-on: ubuntu-18.04
+    env:
+      IMAGE_ARCHIVES_DIR: /home/runner/archives
     steps:
     - name: Checkout code
-      if: github.event_name != 'pull_request' || !github.event.pull_request.head.repo.fork
       uses: actions/checkout@v2
     - name: Docker SSH setup
       if: github.event_name != 'pull_request' || !github.event.pull_request.head.repo.fork
@@ -178,13 +182,30 @@ jobs:
         echo "${{ secrets.DOCKER_PRIVATE_KEY }}" > ~/.ssh/id
         echo "${{ secrets.DOCKER_KNOWN_HOSTS }}" > ~/.ssh/known_hosts
         ssh linkerd-docker docker version
-    - name: Docker build
-      if: github.event_name != 'pull_request' || !github.event.pull_request.head.repo.fork
-      env:
-        DOCKER_HOST: ssh://linkerd-docker
+        echo ::set-env name=DOCKER_HOST::ssh://linkerd-docker
+    - name: Build docker images
       run: |
         export PATH="`pwd`/bin:$PATH"
         DOCKER_TRACE=1 bin/docker-build
+    - name: Create artifact with CLI and image archives
+      if: github.event_name == 'pull_request' && github.event.pull_request.head.repo.fork
+      run: |
+        mkdir -p $IMAGE_ARCHIVES_DIR
+        cp target/cli/linux/linkerd $IMAGE_ARCHIVES_DIR/
+        TAG="$(CI_FORCE_CLEAN=1 bin/root-tag)"
+        for img in controller grafana proxy web ; do
+          docker save "gcr.io/linkerd-io/$img:$TAG" > $IMAGE_ARCHIVES_DIR/$img.tar
+        done
+    # `with.path` values do not support environment variables yet, so an
+    # absolute path is used here.
+    #
+    # https://github.com/actions/upload-artifact/issues/8
+    - name: Upload artifact
+      if: github.event_name == 'pull_request' && github.event.pull_request.head.repo.fork
+      uses: actions/upload-artifact@v1
+      with:
+        name: image-archives
+        path: /home/runner/archives
 
   kind_setup:
     strategy:
@@ -192,13 +213,12 @@ jobs:
       matrix:
         integration_test: [deep, upgrade, helm, helm_upgrade, custom_domain, external_issuer]
     name: Cluster setup (${{ matrix.integration_test }})
+    if: github.event_name != 'pull_request' || !github.event.pull_request.head.repo.fork
     runs-on: ubuntu-18.04
     steps:
     - name: Checkout code
-      if: github.event_name != 'pull_request' || !github.event.pull_request.head.repo.fork
       uses: actions/checkout@v2
     - name: Docker SSH setup
-      if: github.event_name != 'pull_request' || !github.event.pull_request.head.repo.fork
       run: |
         mkdir -p ~/.ssh/
         touch ~/.ssh/id && chmod 600 ~/.ssh/id
@@ -207,7 +227,6 @@ jobs:
         echo "${{ secrets.DOCKER_KNOWN_HOSTS }}" > ~/.ssh/known_hosts
         ssh linkerd-docker docker version
     - name: Kind cluster setup
-      if: github.event_name != 'pull_request' || !github.event.pull_request.head.repo.fork
       env:
         DOCKER_HOST: ssh://linkerd-docker
       run: |
@@ -225,20 +244,27 @@ jobs:
             bin/kind create cluster --name=$KIND_CLUSTER --wait=2m --verbosity 3
         fi
 
-  kind_integration:
+  #
+  # Integration tests that run on the docker host are limited by the fact that
+  # while they all run in separate KinD clusters, they are on the same
+  # machine. The `strategy` context cannot be conditionally set, so there are
+  # separate `kind_integration_*` jobs. This allows the `kind_integration_github`
+  # job to run it's entire matrix in parallel.
+  #
+
+  kind_integration_host:
     strategy:
       max-parallel: 3
       matrix:
         integration_test: [deep, upgrade, helm, helm_upgrade, custom_domain, external_issuer]
     needs: [docker_pull, docker_build, kind_setup]
-    name: Integration tests (${{ matrix.integration_test }})
+    name: Host int. tests (${{ matrix.integration_test }})
+    if: github.event_name != 'pull_request' || !github.event.pull_request.head.repo.fork
     runs-on: ubuntu-18.04
     steps:
     - name: Checkout code
-      if: github.event_name != 'pull_request' || !github.event.pull_request.head.repo.fork
       uses: actions/checkout@v2
     - name: Docker SSH setup
-      if: github.event_name != 'pull_request' || !github.event.pull_request.head.repo.fork
       run: |
         mkdir -p ~/.ssh/
         touch ~/.ssh/id && chmod 600 ~/.ssh/id
@@ -246,8 +272,14 @@ jobs:
         echo "${{ secrets.DOCKER_PRIVATE_KEY }}" > ~/.ssh/id
         echo "${{ secrets.DOCKER_KNOWN_HOSTS }}" > ~/.ssh/known_hosts
         ssh linkerd-docker docker version
+    - name: Try to load cached Go modules
+      uses: actions/cache@v1
+      with:
+        path: ~/go/pkg/mod
+        key: ${{ runner.os }}-go-${{ hashFiles('**/go.sum') }}
+        restore-keys: |
+          ${{ runner.os }}-go-
     - name: Kind load docker images
-      if: github.event_name != 'pull_request' || !github.event.pull_request.head.repo.fork
       run: |
         TAG="$(CI_FORCE_CLEAN=1 bin/root-tag)"
         export KIND_CLUSTER=github-$TAG-${{ matrix.integration_test }}
@@ -262,7 +294,6 @@ jobs:
           done
         EOF
     - name: Install linkerd CLI
-      if: github.event_name != 'pull_request' || !github.event.pull_request.head.repo.fork
       env:
         DOCKER_HOST: ssh://linkerd-docker
       run: |
@@ -275,7 +306,6 @@ jobs:
         [[ "$TAG" == "$($HOME/.linkerd version --short --client)" ]]
         echo "Installed Linkerd CLI version: $TAG"
     - name: Run integration tests
-      if: github.event_name != 'pull_request' || !github.event.pull_request.head.repo.fork
       env:
         DOCKER_HOST: ssh://linkerd-docker
         GITCOOKIE_SH: ${{ secrets.GITCOOKIE_SH }}
@@ -302,18 +332,65 @@ jobs:
           ${{ matrix.integration_test }}_integration_tests
         )
 
+  kind_integration_github:
+    strategy:
+      matrix:
+        integration_test: [deep, upgrade, helm, helm_upgrade, custom_domain, external_issuer]
+    needs: [docker_build]
+    name: GitHub int. tests (${{ matrix.integration_test }})
+    if: github.event_name == 'pull_request' && github.event.pull_request.head.repo.fork
+    runs-on: ubuntu-18.04
+    steps:
+    - name: Checkout code
+      uses: actions/checkout@v2
+    - name: Download image archives
+      uses: actions/download-artifact@v1
+      with:
+        name: image-archives
+    - name: Try to load cached Go modules
+      uses: actions/cache@v1
+      with:
+        path: ~/go/pkg/mod
+        key: ${{ runner.os }}-go-${{ hashFiles('**/go.sum') }}
+        restore-keys: |
+          ${{ runner.os }}-go-
+    - name: Setup KinD (default)
+      if: matrix.integration_test != 'custom_domain'
+      uses: engineerd/setup-kind@v0.3.0
+      with:
+        version: "v0.6.1"
+    - name: Setup KinD (custom_domain)
+      if: matrix.integration_test == 'custom_domain'
+      uses: engineerd/setup-kind@v0.3.0
+      with:
+        config: test/testdata/custom_cluster_domain_config.yaml
+        version: "v0.6.1"
+    - name: Load image archives into KinD
+      run: |
+        for img in controller grafana proxy web; do
+          kind load image-archive image-archives/$img.tar
+        done
+    - name: Install linkerd CLI
+      run: |
+        cp image-archives/linkerd $HOME/.linkerd
+        chmod +x $HOME/.linkerd
+    - name: Run integration tests
+      run: |
+        . bin/_test-run.sh
+        init_test_run $HOME/.linkerd
+        ${{ matrix.integration_test }}_integration_tests
+
   kind_cleanup:
-    if: always()
     strategy:
       fail-fast: false # always attempt to cleanup all clusters
       matrix:
         integration_test: [deep, upgrade, helm, helm_upgrade, custom_domain, external_issuer]
-    needs: [kind_integration]
+    needs: [kind_integration_host]
     name: Cluster cleanup (${{ matrix.integration_test }})
+    if: always() && (github.event_name != 'pull_request' || !github.event.pull_request.head.repo.fork)
     runs-on: ubuntu-18.04
     steps:
     - name: Checkout code
-      if: github.event_name != 'pull_request' || !github.event.pull_request.head.repo.fork
       uses: actions/checkout@v2
     # for debugging
     - name: Dump env
@@ -328,7 +405,6 @@ jobs:
         JOB_CONTEXT: ${{ toJson(job) }}
       run: echo "$JOB_CONTEXT"
     - name: Docker SSH setup
-      if: github.event_name != 'pull_request' || !github.event.pull_request.head.repo.fork
       run: |
         mkdir -p ~/.ssh/
         touch ~/.ssh/id && chmod 600 ~/.ssh/id
@@ -337,7 +413,6 @@ jobs:
         echo "${{ secrets.DOCKER_KNOWN_HOSTS }}" > ~/.ssh/known_hosts
         ssh linkerd-docker docker version
     - name: Kind cluster cleanup
-      if: github.event_name != 'pull_request' || !github.event.pull_request.head.repo.fork
       env:
         DOCKER_HOST: ssh://linkerd-docker
       run: |
@@ -353,14 +428,13 @@ jobs:
 
   docker_deploy:
     name: Docker deploy
+    if: github.ref == 'refs/heads/master' || startsWith(github.ref, 'refs/tags')
     runs-on: ubuntu-18.04
-    needs: [validate_go_deps, go_unit_tests, go_lint, js_unit_tests, kind_integration, kind_cleanup]
+    needs: [validate_go_deps, go_unit_tests, go_lint, js_unit_tests, kind_integration_host, kind_cleanup]
     steps:
     - name: Checkout code
-      if: github.ref == 'refs/heads/master' || startsWith(github.ref, 'refs/tags')
       uses: actions/checkout@v2
     - name: Configure gcloud
-      if: github.ref == 'refs/heads/master' || startsWith(github.ref, 'refs/tags')
       env:
         CLOUDSDK_CORE_DISABLE_PROMPTS: "1"
         CLOUD_SDK_SERVICE_ACCOUNT_KEY: ${{ secrets.CLOUD_SDK_SERVICE_ACCOUNT_KEY }}
@@ -384,7 +458,6 @@ jobs:
         gcloud auth configure-docker
         bin/kubectl version --short
     - name: Docker SSH setup
-      if: github.ref == 'refs/heads/master' || startsWith(github.ref, 'refs/tags')
       run: |
         mkdir -p ~/.ssh/
         touch ~/.ssh/id && chmod 600 ~/.ssh/id
@@ -393,7 +466,6 @@ jobs:
         echo "${{ secrets.DOCKER_KNOWN_HOSTS }}" > ~/.ssh/known_hosts
         ssh linkerd-docker docker version
     - name: Docker push
-      if: github.ref == 'refs/heads/master' || startsWith(github.ref, 'refs/tags')
       env:
         DOCKER_HOST: ssh://linkerd-docker
       run: |
@@ -406,14 +478,13 @@ jobs:
 
   cloud_integration:
     name: Cloud integration tests
+    if: github.ref == 'refs/heads/master' || startsWith(github.ref, 'refs/tags')
     runs-on: ubuntu-18.04
     needs: [docker_deploy]
     steps:
     - name: Checkout code
-      if: github.ref == 'refs/heads/master' || startsWith(github.ref, 'refs/tags')
       uses: actions/checkout@v2
     - name: Configure gcloud
-      if: github.ref == 'refs/heads/master' || startsWith(github.ref, 'refs/tags')
       env:
         CLOUDSDK_CORE_DISABLE_PROMPTS: "1"
         CLOUD_SDK_SERVICE_ACCOUNT_KEY: ${{ secrets.CLOUD_SDK_SERVICE_ACCOUNT_KEY }}
@@ -437,7 +508,6 @@ jobs:
         gcloud auth configure-docker
         bin/kubectl version --short
     - name: Install linkerd CLI
-      if: github.ref == 'refs/heads/master' || startsWith(github.ref, 'refs/tags')
       run: |
         TAG="$(CI_FORCE_CLEAN=1 bin/root-tag)"
         image="gcr.io/linkerd-io/cli-bin:$TAG"
@@ -448,7 +518,6 @@ jobs:
         [[ "$TAG" == "$($HOME/.linkerd version --short --client)" ]]
         echo "Installed Linkerd CLI version: $TAG"
     - name: Run integration tests
-      if: github.ref == 'refs/heads/master' || startsWith(github.ref, 'refs/tags')
       env:
         GITCOOKIE_SH: ${{ secrets.GITCOOKIE_SH }}
       run: |
@@ -457,22 +526,19 @@ jobs:
         version="$($HOME/.linkerd version --client --short | tr -cd '[:alnum:]-')"
         bin/test-run $HOME/.linkerd linkerd-$version
     - name: CNI tests
-      if: github.ref == 'refs/heads/master' || startsWith(github.ref, 'refs/tags')
       run: |
         export TAG="$($HOME/.linkerd version --client --short)"
         go test -cover -race -v -mod=readonly ./cni-plugin/test -integration-tests
 
   cloud_cleanup:
-    if: always()
+    if: always() && (github.ref == 'refs/heads/master' || startsWith(github.ref, 'refs/tags'))
     name: Cloud cleanup
     runs-on: ubuntu-18.04
     needs: [cloud_integration]
     steps:
     - name: Checkout code
-      if: github.ref == 'refs/heads/master' || startsWith(github.ref, 'refs/tags')
       uses: actions/checkout@v2
     - name: Configure gcloud
-      if: github.ref == 'refs/heads/master' || startsWith(github.ref, 'refs/tags')
       env:
         CLOUDSDK_CORE_DISABLE_PROMPTS: "1"
         CLOUD_SDK_SERVICE_ACCOUNT_KEY: ${{ secrets.CLOUD_SDK_SERVICE_ACCOUNT_KEY }}
@@ -495,7 +561,6 @@ jobs:
         gcloud auth configure-docker
         bin/kubectl version --short
     - name: Cleanup
-      if: github.ref == 'refs/heads/master' || startsWith(github.ref, 'refs/tags')
       run: |
         export PATH="`pwd`/bin:$PATH"
         bin/test-cleanup
@@ -507,14 +572,13 @@ jobs:
 
   chart_deploy:
     name: Helm chart deploy
+    if: startsWith(github.ref, 'refs/tags')
     runs-on: ubuntu-18.04
     needs: [cloud_integration]
     steps:
     - name: Checkout code
-      if: startsWith(github.ref, 'refs/tags')
       uses: actions/checkout@v2
     - name: Configure gsutils
-      if: startsWith(github.ref, 'refs/tags')
       env:
         CLOUDSDK_CORE_DISABLE_PROMPTS: "1"
         CLOUD_SDK_SERVICE_ACCOUNT_KEY: ${{ secrets.LINKERD_SITE_TOKEN }}