Commit Graph

118 Commits

Author SHA1 Message Date
Joakim Roubert 5c104ebec6
Run shellcheck for all shell scripts in repository (#4441)
* Run shellcheck for all shell scripts in repository

Update the shellcheck command in static_checks.yml to not only scan the
contents of ./bin, but search for all files with mimetype
text/x-shellscript and feed them to shellcheck.

Certainly, this is a tad more time consuming than just scanning one
directory, but still a quite fast thing to do while it prevents any
new scripts to fly under the radar.

(Also, there is no need to exclude *.nuspec or *.ps1 from the find
command as they do not have the text/x-shellscript mimetype.)

Change-Id: I7433d231e8a315df65c03ee8765914e782057343
Signed-off-by: Joakim Roubert <joakimr@axis.com>

* Updates after review comment

Move shellcheck of all scripts to own script that is then called by
static_checks.yml as suggested by @kleimkuhler.
Also updated sources for helm-build and kind-load so that the
new shellcheck-all script can be called from any directory.

Change-Id: I9e82230459cb843c4143ec979c93060f424baed8
Signed-off-by: Joakim Roubert <joakim.roubert@axis.com>
2020-05-20 14:08:45 -07:00
Alejandro Pedraza 301429ea9b
Bump KinD to 0.8.1 (#4445)
* Bump KinD to 0.8.1

This brings us K8s 1.18, which is in theory passing all the integration
tests. Currently the tracing one is failing just because of the quay.io
downtime, that hosts the nginx-ingress image.

Re #4382
2020-05-20 14:46:05 -05:00
Joakim Roubert 960ce556ba
bin/_log.sh: Add shebang to please shellcheck (#4437)
Signed-off-by: Joakim Roubert <joakimr@axis.com>
2020-05-20 09:55:51 -07:00
Joakim Roubert ef67cbed38
bin/lint: Fix shellcheck issue (#4434)
Delete variable `os` that is not used. The golangci-lint downloader script does its own extensive platform lookup before downloading the selected binary.

Signed-off-by: Joakim Roubert <joakimr@axis.com>
2020-05-19 23:23:25 -07:00
Kevin Leimkuhler d99c1486ba
Lint all markdown files in CI (#4402)
## Motivation

linkerd/rfc#22

## Solution

Use the [markdown-lint-action](https://github.com/marketplace/actions/markdown-linting-action) to lint all `.md` files for all pull requests
and pushes to master.

This action uses the default rules outlined in [markdownlint
package](https://github.com/DavidAnson/markdownlint/blob/master/doc/Rules.md).

The additional rules are added are explained below:
- Ignore line length lints for code blocks
- Ignore line length lints for tables
- Allow duplicate sub-headers in sibling headers (e.g. allowing multiple ##
  Significant headers in `CHANGES.md` as long as they are part of separate
  release headers)

Signed-off-by: Kevin Leimkuhler <kevin@kleimkuhler.com>
2020-05-19 23:03:50 -07:00
Joakim Roubert 30ba9a1261
bin/fmt: Fix shellcheck issue (#4438)
Signed-off-by: Joakim Roubert <joakimr@axis.com>
2020-05-19 14:49:28 -07:00
Joakim Roubert 6f1654a65d
bin/_tag.sh: Fix shellcheck issues (#4436)
Signed-off-by: Joakim Roubert <joakimr@axis.com>
2020-05-19 14:49:07 -07:00
Joakim Roubert b2082712b5
bin/update-go-deps-shas: Fix shellcheck issues (#4435)
Pass grep output through xargs.
Use `${0%/*}` instead of `$bindir `since the variable `bindir` exists in
_tag.sh too and then triggers the shellcheck variable modifed warning.
Script uses no bash features and can thus be a POSIX /bin/sh script.

Signed-off-by: Joakim Roubert <joakimr@axis.com>
2020-05-19 13:00:34 -07:00
Joakim Roubert 406107bc87
bin/_docker.sh: Fix shellcheck issues (#4433)
Signed-off-by: Joakim Roubert <joakimr@axis.com>
2020-05-19 10:39:41 -07:00
Joakim Roubert 56484ade8d
bin/test-clouds: Fix shellcheck issues (#4423)
Signed-off-by: Joakim Roubert <joakimr@axis.com>
2020-05-18 13:42:46 -07:00
Joakim Roubert 3ef358bb2f
bin/protoc-go.sh: Fix shellcheck error (#4420)
Signed-off-by: Joakim Roubert <joakimr@axis.com>
2020-05-18 13:13:41 -07:00
Joakim Roubert 68e25f2c11
bin/test-clouds-cleanup: Fix shellcheck issues (#4422)
shellcheck will not accept the string DO since it is not sure whether it is a misspelled do command or a string with DO. Explicitly quoting it will mitigate this.

Signed-off-by: Joakim Roubert <joakimr@axis.com>
2020-05-18 11:53:24 -07:00
Joakim Roubert cc1279b4ba
Fix SC1090 shellcheck issues in shell script files (#4417)
The SC1090 "Can't follow non-constant source" issue is addressed in the way suggested in shellcheck's documentation; the source paths are pointed out in shellcheck comments. By adding the bin dir to the -P shellcheck CLI parameter, we avoid having to state the bin directory in each and every script file.

Signed-off-by: Joakim Roubert <joakimr@axis.com>
2020-05-18 11:32:14 -07:00
Joakim Roubert 55326a61ac
bin/web: Fix shellcheck issues (#4425)
Signed-off-by: Joakim Roubert <joakimr@axis.com>
2020-05-18 10:46:28 -07:00
Joakim Roubert 9c639dc3b7
bin/test-scale: Fix shellcheck issues (#4424)
Remove superfluous echo commands in assignments.
Add quotes.
Simplify the for loops that shellcheck didn't like.

Signed-off-by: Joakim Roubert <joakimr@axis.com>
2020-05-18 10:41:49 -07:00
Joakim Roubert 5eba710f54
bin/mkube: Update according to shellcheck suggestions (#4419)
Also clean up sed Windows path filtering.

Signed-off-by: Joakim Roubert <joakimr@axis.com>
2020-05-18 10:03:42 -07:00
Alejandro Pedraza 8b0122bf94
Refactor CNI integration tests to use annotations functions (#4363)
Followup to #4341

Replaced all the `t.Error`/`t.Fatal` calls in the integration tests with the
new functions defined in `testutil/annotations.go` as described in #4292,
in order for the errors to produce Github annotations.

This piece takes care of the CNI integration test suite.

This also enables the annotations for these and the general integration
tests, by setting the `GH_ANNOTATIONS` environment variable in the
workflows whose flakiness we're interested on catching: Kind
integration, Cloud integration and Release.

Re #4176
2020-05-14 12:13:07 -05:00
Alejandro Pedraza a35ad28e14
Use gotestsum in CI unit test workflow (#4342)
Use [gotestsum](https://github.com/gotestyourself/gotestsum) for running
unit tests in CI, so we get a summary result at the end, instead of having to
scroll up to find failures.

Doesn't apply for integration tests, as only failures are shown there,
and they're easily visible.
2020-05-07 08:56:58 -05:00
Alejandro Pedraza 9e9f3bb1e2
Not use preemptible nodes in cloud and release workflows (#4315)
Temporarily set `preemptible: false` for cloud and release workflows to
see if that's the cause of nodes getting killed.
2020-04-30 18:05:24 -05:00
drholmie 7a560a723d
Linkerd CLI Chocolatey Package (#4205)
* Add Linkerd CLI Chocolatey Package

This PR partially fixes #3063 by building a chocolatey package for Linkerd2's Windows CLI
It adds the build scripts for the Linkerd chocolatey package and based on discussions in
https://github.com/linkerd/linkerd2/pull/3921

Signed-off-by: Animesh Narayan Dangwal <animesh.leo@gmail.com>
2020-04-29 09:41:54 -07:00
Alejandro Pedraza 66ec92aa09
Additional Jest reporter for GH Annotations (#4294)
Second part of #4176

Added extra Jest reporter when running js tests from CI, which will send
to stdout a GH annotation for each test failure, something like:

```
::error file=/home/alpeb/src/forks/linkerd2/web/app/js/components/Navigation.test.jsx::Navigation › checks state when versions do not match
```

See the [health
metrics RFC](https://github.com/linkerd/rfc/blob/master/design/0002-ci-health-metrics.md) for more context.
2020-04-28 13:10:27 -05:00
Kevin Leimkuhler c8bca67306
Update integration tests to use kind 0.7.0 (#4282)
I missed these integration test version bumps in #4280 

Signed-off-by: Kevin Leimkuhler <kevin@kleimkuhler.com>
2020-04-23 13:53:00 -07:00
Thomas Rampelberg 426d558f43
Fix security issues by bumping all dependencies (#4284)
* Fix security issues by bumping all dependencies.

* Bump node version to latest
2020-04-23 10:31:55 -07:00
Alejandro Pedraza 322ba5fd2f
`linkerd uninstall` errors when attempting to delete PSP (#4234)
* Bug in `linkerd uninstall` when attempting to delete PSP

We were using a wrong apiVersion for PSP in `linkerd uninstall`'s
output, which avoids removing that resource:

```
$ linkerd uninstall | kubectl delete -f -
clusterrole.rbac.authorization.k8s.io "linkerd-linkerd-controller"
deleted
clusterrole.rbac.authorization.k8s.io "linkerd-linkerd-destination"
deleted
...
mutatingwebhookconfiguration.admissionregistration.k8s.io
"linkerd-proxy-injector-webhook-config" deleted
validatingwebhookconfiguration.admissionregistration.k8s.io
"linkerd-sp-validator-webhook-config" deleted
namespace "linkerd" deleted
error: unable to recognize "uninstall.yml": no matches for kind
"PodSecurityPolicy" in version "extensions/v1beta1"

$ kubectl get psp -oname
podsecuritypolicy.policy/linkerd-linkerd-control-plane
```

I've also replaced the uninstall integration test with a new separate
suite that performs the installation, waits for it to be ready,
uninstalls, and then confirms `linkerd check --pre` returns as expected.
2020-04-07 11:01:11 -05:00
Alejandro Pedraza be558b6869
Don't include git SHA in cloud_integration_tests names - part 2 (#4231)
Followup to #4230

I forgot to make the same change to the `release.yml` workflow
2020-04-02 19:43:27 -05:00
Alejandro Pedraza 84a9e2a807
Don't include git SHA in cloud_integration_tests namespaces (#4230)
The `cloud_integration_tests` job was creating its tests under
namespaces containing the git SHA. This is a left-over from when all the
tests ran in the same cluster, which is no longer the case, and thus no
longer needed.

This fixes the [current CI
failure](https://github.com/linkerd/linkerd2/runs/556330879?check_suite_focus=true#step:6:24)
in master.
2020-04-02 18:22:00 -05:00
Alejandro Pedraza 0c8171d466
Fix bin/kind-load for pull requests (#4222)
* Fix bin/kind-load for pull requests

Followup to #4212

External PRs were failing because:

1) The image tarballs weren't being loaded from the `images-archives`
directory
2) Concurrent calls to `bin/kind` were attempting to download the KinD
binary simultaneously, resulting in a "text file busy" error. To avoid
that, now we just call `bin/kind` synchronously one time beforehand.
2020-04-01 12:04:24 -05:00
Alejandro Pedraza 22f1606b73
Extract common logic in scripts and CI to load images into KinD (#4212)
Fixes #4206 Followup to #4167

Extract common logic to load images into KinD, from `bin/kind-load`, `bin/install-pr`, `.github/workflows/kind_integration.yml` and `.github/workflows/release.yml`.

Besides removing the duplication, `bin/kind-load` will benefit in performance by having each image be loaded in parallel.

```
Load into KinD the images for Linkerd's proxy, controller, web, grafana, debug and cni-plugin.

Usage:
    bin/kind-load [--images] [--images-host ssh://linkerd-docker]

Examples:

    # Load images from the local docker instance
    bin/kind-load

    # Load images from tar files located in the current directory
    bin/kind-load --images

    # Retrieve images from a remote docker instance and then load them into KinD
    bin/kind-load --images --images-host ssh://linkerd-docker

Available Commands:
    --images: use 'kind load image-archive' to load the images from local .tar files in the current directory.
    --images-host: the argument to this option is used as the remote docker instance from which images are first retrieved
                   (using 'docker save') to be then loaded into KinD. This command requires --images.
```
2020-03-30 16:28:28 -05:00
Kevin Leimkuhler 88cafa36c6
Upload artifacts for all PRs (#4159)
## Motivation

#4147 adds a script for setting up a local cluster that uses the images built
from the changes introduced in a forked PR. This would be useful for all PRs.

In order to install Linkerd from a PR into a local cluster, the images still
need to be built at some point. If you happen to have SSH config setup for our
Packet host, you can pull them from there. That is not very
accessible--requiring that someone adds you as a user--so we can take a
similar approach to forked PRs.

## Solution

All PRs now make an artifact directory that is uploaded as part of the KinD
integration tests. This way, the `install-pr` script can use those images no
matter if the PR is a fork or not.
2020-03-10 12:44:27 -07:00
cpretzer abbf4a4e60
Set the published version check to sleep for 30 seconds (#4143)
Signed-off-by: Charles Pretzer <charles@buoyant.io>
2020-03-06 12:27:21 -08:00
Alejandro Pedraza 578a2d1960
CI: Adjustments to the release job (#4129)
Extracted the logic to pull the latest release notes, out of
`bin/create-release-tag` into `bin/_release.sh` so that it can be reused
in the `release.yml` workflow, which needs to use that inside
`gh_release` when creating the github release in order to have prettier
markup release notes instead of a plaintext message pulled out of the tag
message.
The new extracted function also receives an optional argument with the
name of the file to put the release notes into, because the `body_path`
parameter in `softprops/action-gh-release` doesn't work with dynamic
vars.

Finally, now the `website_publish` job will only launch until the `gh_release`
has succeeded.
2020-03-05 09:03:30 -05:00
Andrew Seigner a37316a336
Introduce `bin/shellcheck`, add to ci (#4118)
PR #4117 was root-caused with the help of `shellcheck`.

This change introduces a `bin/shellcheck` script, and adds it to CI. In
CI, many checks are disabled to allow it to pass. This will at least
prevent introduction of new classes of shell issue, and should motivate
re-enabling more checks over time.

Signed-off-by: Andrew Seigner <siggy@buoyant.io>
2020-03-02 13:18:08 -08:00
Alejandro Pedraza a65f76ed22
Use SHAs instead of tags when referring to GH Actions libs (#4114)
When adding an action we can quickly vet it and fix it to a sha. Whereas
if we use a tag, the 3rd party can change the code and retag it without us noticing
2020-03-02 15:03:24 -05:00
Alejandro Pedraza 43e64e4818
Added last release job to verify version was published in the website (#4106)
Added the `website_publish_check` job, triggered upon edge/stable tag pushes,
that verifies that the install script at the website has indeed been
updated with the corresponding edge/stable version. It performs the
check every 5 seconds, 10 times, before giving up and failing the build
run. Tested fine in my fork 👍
2020-02-26 15:03:45 -05:00
Kevin Leimkuhler ae880f0e33
Create linkerd/website dispatch event on release (#4100)
## Motivation

A goal of the release automation project is to automate the website publish
that publishes a new install script that uses the new release version.

linkerd/website#668 removed the hard coded versions from the repo and moved
the version update into the `make publish` command.

That workflow now needs to be triggered by a release in `linkerd2`.

## Solution

Once `kind_integration_tests` and `cloud_integration_test` pass, a job runs
using the [repository-dispatch](https://github.com/marketplace/actions/repository-dispatch) action to create a repository dispatch event in
`linkerd/website`.

This dispatch event will (current PR: linkerd/website#670) trigger the
publish workflow.

## Testing

Tested in my fork [here](https://github.com/kleimkuhler/linkerd2/actions/runs/45165789)

## Additional steps needed

A new `RELEASE_TOKEN` secret needs to be added to this repo. It should be the
personal access token of [l5d-bot](https://github.com/l5d-bot) with `repo` access.

Signed-off-by: Kevin Leimkuhler <kevin@kleimkuhler.com>
2020-02-26 09:42:23 -08:00
Alejandro Pedraza c355ae8ff0
New CI job to automatically generate the Github release (#4094)
* New CI job to automatically generate the Github release

Fixes #4083

New `gh_release` job in the `release.yml` for creating the release in
Github and uploading the CLI binaries for each platform, along with
their checksum files.

This job only gets triggered upon successful docker images building and
pushing, and kind and cloud integration tests passing.

The Helm chart deploying job gets now triggered upon the success of this
new job.

Signed-off-by: Alejandro Pedraza <alejandro@buoyant.io>
2020-02-26 09:05:55 -05:00
Alejandro Pedraza 2ad141d27a
Exclude changes on markup files to trigger CI runs in master (#4084)
Fixes #4082

This tested fine in a fork, under various scenarios combining:
- Modify markup file in root dir
- Modify markup file in subdir
- Modify non-markup file
- In master
- In a PR
2020-02-24 13:50:35 -05:00
Kevin Leimkuhler ab4a13ab52
Add minimal release workflow (#4090)
## Motivation

A release workflow will be the only triggered workflow on `push.tags` events.

As a first step in automating the release process, it should assert that
integration tests pass once the docker images have been tagged.

Both KinD and cloud integration tests should run since they have different
sets of integration tests that they are responsible for running.

It then needs to run the `chart_deploy` job.

## Testing

This has been fully tested with a release tag push on my fork. The run can be
found [here](https://github.com/kleimkuhler/linkerd2/actions/runs/42664128)

It properly failed on `chart_deploy` because I did not want to push a test tag
helm chart.

## Solution

This workflow will:

- Build the docker images on the Packet Host
- Tag the docker images with the release tag
- Run KinD integration tests
- Run cloud integration tests
- Run `chart_deploy`

Signed-off-by: Kevin Leimkuhler <kevin@kleimkuhler.com>
2020-02-22 17:13:48 -08:00
Alejandro Pedraza 8c12f03af8
Update gcloud configs for the chart_deploy job - Part 2 (#4087) 2020-02-20 18:10:40 -05:00
Alejandro Pedraza 262fe578b6
Update gcloud configs for the chart_deploy job (#4086) 2020-02-20 17:00:52 -05:00
Alejandro Pedraza df2011dbb2
CI: Upgrades the gcloud action to fix GKE clusters teardown issue (#4074)
Ref linkerd/linkerd2-action-gcloud#1
2020-02-19 17:24:59 -05:00
Kevin Leimkuhler c31284e6db
Separate single Actions workflow into multiple workflows (#4039)
Depends on #4033

## Motivation

If any job fails in the current GH Actions workflow, a re-run on the same
commit SHA requires re-running *all* jobs--regardless if the job already
passed in the previous run.

This can be problematic when dealing with flakiness in the integration tests.

If a test fails due to flakiness in `cloud_integration_tests`, all the unit
tests, static checks, and `kind_integration_tests` are re-run which leads to
lots of waiting and dealing with the possibility of flakiness in earlier jobs.

With this change, individual workflows can now be re-run without triggering
all other jobs to complete again first.

## Solution

`workflow.yml` is now split into:
- `static_checks.yml`
- `unit_tests.yml`
- `kind_integration.yml`
- `cloud_integration.yml`

### Workflows

`statc_checks.yml` performs checks related to dependencies, linting, and
formatting.

`unit_tests.yml` performs the Go and JS unit tests.

`kind_integration.yml` builds the images (on Packet or the GH Action VM) and
runs the integration tests on a KinD cluster. This workflow continues to run
for **all** PRs and pushes to `master` and tags.

`cloud_integration.yml` builds the images only on Packet. This is because
forked repositories do not need to trigger this workflow. It then creates a
unique GKE cluster and runs the integration tests on the cluster.

### The actual flow of work..

A forked repository or non-forked repository opening a PR triggers:
- `static_checks`
- `unit_tests`
- `kind_integration_tests`

These workflows all run in parallel and are invidivually re-runnable.

A push to `master` or tags triggers:
- `static_checks`
- `unit_tests`
- `kind_integration_tests`
- `cloud_integration_tests`

These workflows also all run in parallel, including the `docker_build` step of
both integration test workflows. This has not conflicted in testing as it
takes place on the same Packet host and just utilizes docker layer caching
well.

Signed-off-by: Kevin Leimkuhler <kevin@kleimkuhler.com>
2020-02-13 09:11:30 -08:00
Kevin Leimkuhler a460ada166
Run all PRs on GH Actions VMs (#4033)
Run all PRs on GH Actions VMs

## Motivation

Currently all pushes to master branch, tags, and Linkerd org member PRs run
the `kind_integration_host` job on the same Packet host.

The means that parallel jobs spin up KinD clusters with a unique name and
sandbox the tests so that they do not clash.

This is problematic for a few reasons:
* There is a limit on the number of jobs we can run in parallel due to
  resource constraints.
* Workflow cancellation and re-runs conflict when the cancelled run deletes
  it's namespaces and the running one expects them to be present.
* There has been an observed flakiness with running multiple KinD clusters
  resulting in inconsistent timeouts and docker errors.

## Solution

This change moves all KinD integration testing to GH Actions VMs. This is
currently what forked repository workflows do.

There is no longer a `docker_pull` job as it's responsibilities has been moved
into one of the `kind_integration_tests` steps.

The renamed `kind_integration_tests` job is responsible for **all** PR
workflows and has steps specific to forked and non-forked repositories.

### Non-forked repository PRs

The Packet host is still used for building docker images as leveraging docker
layer caching is still valuable--a build can be as fast as 30 seconds compared
to ~12 minutes.

Loading the docker images into the KinD cluster on the GH Action VM is done by
saving the Packet host docker images as image archives, and loading those
directly into the local KinD cluster.

### Forked repository PRs

`docker_build` has been sped up slightly by sending `docker save` processes to
the background.

Docker layer caching cannot be leveraged since there are no SSH secrets
available, so the `artifact-upload`/`artifact-download` actions introduced in
#TODO are still used.

### Cleanup

This PR also includes some general cleanup such as:
* Some job names have been renamed to better reflect their purpose or match
  the current naming pattern.
* Environment variables are set earlier in jobs as a separate step if it is
  currently exported multiple times.
* Indentation was really bothering me because it switches back and forth
  throughout the workflow file, so lists are now indented.

Signed-off-by: Kevin Leimkuhler <kevin@kleimkuhler.com>
2020-02-12 14:38:05 -08:00
Alejandro Pedraza 1e8223e143
Allow CI to run concurrent builds in master (#4001)
* Allow CI to run concurrent builds in master

Fixes #3911

Refactors the `cloud_integration` test to run in separate GKE clusters
that are created and torn down on the fly.
It leverages a new "gcloud" github action that is also used to set up
gcloud in other build steps (`docker_deploy` and `chart_deploy`).

The action also generates unique names for those clusters, based on the
git commit SHA and `run_id`, a recently introduced variable that is
unique per CI run and available to all the jobs.
This fixes part of #3635 in that CI runs on the same SHA don't interfere
with one another (in the `cloud_integration` test; still to do for
`kind_integration`).

The "gcloud" GH action is hosted under its own repo in https://github.com/linkerd/linkerd2-action-gcloud
2020-02-07 16:23:36 -05:00
Kevin Leimkuhler ae7d98b4fe
Run integration tests for forked repos (#4002)
* Allow CI to run concurrent builds in master

Fixes #3911

Refactors the `cloud_integration` test to run in separate GKE clusters
that are created and torn down on the fly.
It leverages a new "gcloud" github action that is also used to set up
gcloud in other build steps (`docker_deploy` and `chart_deploy`).

The action also generates unique names for those clusters, based on the
git commit SHA and `run_id`, a recently introduced variable that is
unique per CI run and available to all the jobs.
This fixes part of #3635 in that CI runs on the same SHA don't interfere
with one another (in the `cloud_integration` test; still to do for
`kind_integration`).

The "gcloud" GH action is supported by `.github/actions/gcloud/index.js`
that has a couple of dependencies. To avoid having to commit
`node_modules`, after every change to that file one must run
```bash
# only needed the first time
npm i -g @zeit/ncc

cd .github/actions/gcloud
ncc build index.js
```
which generates the self-contained file
`.github/actions/gcloud/dist/index.js`.
(This last part might get easier in the future after other refactorings
outside this PR).

* Run integration tests for forked repos

Signed-off-by: Kevin Leimkuhler <kevin@kleimkuhler.com>

* Address reviews

Signed-off-by: Kevin Leimkuhler <kevin@kleimkuhler.com>

* Address more reviews

Signed-off-by: Kevin Leimkuhler <kevin@kleimkuhler.com>

* Move some conditionals to jobs

Signed-off-by: Kevin Leimkuhler <kevin@kleimkuhler.com>

* Change job name

Signed-off-by: Kevin Leimkuhler <kevin@kleimkuhler.com>

* Move more conditionals to job level

Signed-off-by: Kevin Leimkuhler <kevin@kleimkuhler.com>

* Added more flags to 'gcloud container clusters create' and consolidated
'create' and 'destroy' into ' action'

* Run kind cleanup only for non-forked PRs

Signed-off-by: Kevin Leimkuhler <kevin@kleimkuhler.com>

* Got rid of cloud_cleanup by using a post hook in the gcloud action

* Removed cluster naming responsibility from the gcloud action

* Consolidate .gitignore statements

* Removed bin/_gcp.sh

* Change name of Kind int. test job

Signed-off-by: Kevin Leimkuhler <kevin@kleimkuhler.com>

* Ensure `kind_cleanup` still runs on cancelled host CI runs

Signed-off-by: Kevin Leimkuhler <kevin@kleimkuhler.com>

* Add reviews

Signed-off-by: Kevin Leimkuhler <kevin@kleimkuhler.com>

* Update workflow comment

Signed-off-by: Kevin Leimkuhler <kevin@kleimkuhler.com>

* Split index.js into setup.js and destroy.js

* trigger build

* Moved the gcloud action into its own repo

* Full version for the gcloud GH action

* Rebase back to master

Signed-off-by: Kevin Leimkuhler <kevin@kleimkuhler.com>

* Remvoe additional changes

Signed-off-by: Kevin Leimkuhler <kevin@kleimkuhler.com>

* Remove additional changes

Signed-off-by: Kevin Leimkuhler <kevin@kleimkuhler.com>

* Trigger CI

Signed-off-by: Kevin Leimkuhler <kevin@kleimkuhler.com>

Co-authored-by: Alejandro Pedraza <alejandro.pedraza@gmail.com>
2020-02-07 12:27:04 -08:00
Zahari Dichev c609564dc8
Add helm upgrade integration test (#3976)
In light of the breaking changes we are introducing to the Helm chart and the convoluted upgrade process (see linkerd/website#647) an integration test can be quite helpful. This simply installs latest stable through helm install and then upgrades to the current head of the branch.

Signed-off-by: Zahari Dichev zaharidichev@gmail.com
2020-02-04 08:27:46 +02:00
Kevin Leimkuhler 53baecb382
Changes for edge-20.1.3 (#3966)
## edge-20.1.3

* CLI
  * Introduced `linkerd check --pre --linkerd-cni-enabled`, used when the CNI
    plugin is used, to check it has been properly installed before proceeding
    with the control plane installation
  * Added support for the `--as-group` flag so that users can impersonate
    groups for Kubernetes operations (thanks @mayankshah160!)
* Controller
  * Fixed an issue where an override of the Docker registry was not being
    applied to debug containers (thanks @javaducky!)
  * Added check for the Subject Alternate Name attributes to the API server
    when access restrictions have been enabled (thanks @javaducky!)
  * Added support for arbitrary pod labels so that users can leverage the
    Linkerd provided Prometheus instance to scrape for their own labels
    (thanks @daxmc99!)
  * Fixed an issue with CNI config parsing

Signed-off-by: Kevin Leimkuhler <kevin@kleimkuhler.com>
2020-01-23 16:55:21 -08:00
Alejandro Pedraza aa619ed87c
Upgraded actions/checkout to v2 (#3889)
* Revert "Replace actions/checkout with actions/upload/download (#3602)"

This reverts commit 397970e917.

* Upgraded actions/checkout to @v2

Reverts #3602 and Fixes #3881
2020-01-08 17:57:36 -05:00
Alejandro Pedraza 2c0b6efc17
Use `kind export kubeconfig` instead of `scp` (#3875)
* Use `kind export kubeconfig` instead of `scp`

Followup to #3864
[comment](https://github.com/linkerd/linkerd2/pull/3864#discussion_r360976473)

Stop moving the kubeconfig file between the Github Action env and the
build server with `scp` and use instead `kind export kubeconfig`.

* Replaced deprecated '--loglevel debug' flag with '--verbosity 3'
2020-01-06 18:54:59 -05:00
Alejandro Pedraza bb790b22b4
Upgrade `kind` to v0.6.1 (#3864)
* Upgrade `kind` to v0.6.1

Fixes #3852

Upgraded `/bin/kind` to pull v0.6.1.
Also have `workflow.yml` use `KUBECONFIG` explicitly for setting the
location of the config file, now that `kind get kubeconfig-path` has
been deprecated (check
https://github.com/kubernetes-sigs/kind/releases/tag/v0.6.0 for detailed
info).
Note that in the build server the kind binary for this version is
`kind-0.6.1`, leaving the `kind` binary still pointing to v0.5.1 while
this gets merged and all the PR branches get this.
2019-12-30 14:32:37 -05:00
Alejandro Pedraza 602dad8d33
When running `kind load` in the build host, don't ignore `ssh`'s stderr (#3824)
* When running `kind load` in the build host, don't ignore `ssh`'s stderr
2019-12-20 09:45:56 -05:00
Paul Balogh 2cd2ecfa30 Enable mixed configuration of skip-[inbound|outbound]-ports (#3766)
* Enable mixed configuration of skip-[inbound|outbound]-ports using port numbers and ranges (#3752)
* included tests for generated output given proxy-ignore configuration options
* renamed "validate" method to "parseAndValidate" given mutation
* updated documentation to denote inclusiveness of ranges
* Updates for expansion of ignored inbound and outbound port ranges to be handled by the proxy-init rather than CLI (#3766)

This change maintains the configured ports and ranges as strings rather than unsigned integers, while still providing validation at the command layer.

* Bump versions for proxy-init to v1.3.0

Signed-off-by: Paul Balogh <javaducky@gmail.com>
2019-12-20 09:32:13 -05:00
Carol A. Scott 857a03c79c
Update yarn and add network-concurrency flag (#3826)
This PR addresses recent JS unit test failures on CI by: 

* Upgrading yarn from 1.7.0 to 1.21.1 (current stable version) in the Dockerfile 
and Github Actions workflow 
* Wrapping the yarn installation with the --network-concurrency 1 flag, setting the 
maximum number of concurrent network requests to 1, suggested as a fix here:
https://github.com/yarnpkg/yarn/issues/2629
2019-12-13 16:49:52 -08:00
Andrew Seigner c69080ee97
GitHub Actions: known_hosts, max-parallel, gcloud (#3733)
Follow up from #3730, GitHub Actions was failing in the following ways:
- booting 5 parallel kind clusters in the matrix job was overloading the
  Docker host
- `~/.ssh/known_hosts` file was empty following gcloud sdk installation

Four changes to mitigate these issues:
- generate `known_hosts` from a secret, rather than an `ssh-keyscan`
  command
- generate `~/.ssh/config` from a secret
- set `max-parallel: 3` on the matrix job
- install gcloud sdk prior to Docker ssh setup

TODO:
- remove `DOCKER_ADDRESS`, `DOCKER_HOST_PRIVATE_KEY`

Signed-off-by: Andrew Seigner <siggy@buoyant.io>
2019-11-15 17:19:13 -08:00
Zahari Dichev 2d224302de
Add integration test for external issuer and cert rotation flows (#3709)
Signed-off-by: zaharidichev <zaharidichev@gmail.com>
2019-11-14 06:58:32 +02:00
Alejandro Pedraza 3324966702
Upgrade go to 1.13.4 (#3702)
Fixes #3566

As explained in #3566, as of go 1.13 there's a strict check that ensures a dependency's timestamp matches it's sha (as declared in go.mod). Our smi-sdk dependency has a problem with that that got resolved later on, but more work would be required to upgrade that dependency. In the meantime a quick pair of replace statements at the bottom of go.mod fix the issue.
2019-11-13 12:54:36 -05:00
Zahari Dichev 1bb9d66757 Integration test for custom cluster domain (#3660)
Signed-off-by: zaharidichev <zaharidichev@gmail.com>
2019-11-04 14:49:52 -08:00
Alejandro Pedraza 397970e917
Replace actions/checkout with actions/upload/download (#3602)
* Replace actions/checkout with actions/upload/download

Signed-off-by: Alejandro Pedraza Borrero <alejandro@buoyant.io>
2019-10-23 17:22:59 -05:00
Alejandro Pedraza e76c5c3d9d
Keep old releases in Helm repo index (#3589)
* Keep old releases in Helm repo index

When building the Helm repo index file, keep the references to the old
releases. Also rename and keep the old index file in case
something goes wrong when generating the new one.

Fixes #3561
2019-10-16 17:21:53 -05:00
Alex Leong 3dcff52b9f
Switch from using golangci fmt to using goimports (#3555)
CI currently enforcing formatting rules by using the fmt linter of golang-ci-lint which is invoked from the bin/lint script.  However it doesn't seem possible to use golang-ci-lint as a formatter, only as a linter which checks formatting.  This means any formatter used by your IDE or invoked manually may or may not use the same formatting rules as golang-ci-lint depending on which formatter you use and which specific revision of that formatter you use.  

In this change we stop using golang-ci-lint for format checking.  We introduce `tools.go` and add goimports to the `go.mod` and `go.sum` files.  This allows everyone to easily get the same revision of goimports by running `go install -mod=readonly golang.org/x/tools/cmd/goimports` from inside of the project.  We add a step in the CI workflow that uses goimports via the `bin/fmt` script to check formatting.

Some shell gymnastics were required in the `bin/fmt` script to work around some limitations of `goimports`:
* goimports does not have a built-in mechanism for excluding directories, and we need to exclude the vendor director as well as the generated Go sources
* goimports returns a 0 exit code, even when formatting errors are detected

Signed-off-by: Alex Leong <alex@buoyant.io>
2019-10-16 13:56:11 -07:00
Alejandro Pedraza 6764d03e52
Tiny: remove uinintended comment (#3425)
Followup to #3415

Signed-off-by: Alejandro Pedraza <alejandro@buoyant.io>
2019-09-12 12:56:46 -05:00
Alejandro Pedraza 8270ba363c
Add chart_deploy into workflow.yml (#3415)
* Have CI push the Helm artifacts into GCS

- Added missing OWNERS and README files
- Added maintainers section to  Chart.yaml
- Changed NOTES.txt so it points to the installation of the CLI
- Set the proxy-init version to v1.1.0 in values.yaml

Ref #3256

Signed-off-by: Alejandro Pedraza <alejandro@buoyant.io>
2019-09-11 12:09:50 -05:00
Andrew Seigner c5a85e587c
Update to client-go v12.0.0, forked stern (#3387)
The repo depended on an old version of client-go. It also depended on
stern, which itself depended on an old version of client-go, making
client-go upgrade non-trivial.

Update the repo to client-go v12.0.0, and also replace stern with a
fork.

This fork of stern includes the following changes:
- updated to use Go Modules
- updated to use client-go v12.0.0
- fixed log line interleaving:
  - https://github.com/wercker/stern/issues/96
  - based on:
    - 8723308e46

Fixes #3382

Signed-off-by: Andrew Seigner <siggy@buoyant.io>
2019-09-10 11:04:29 -07:00
Andrew Seigner 46d33a10e4
Preload Prometheus and proxy-init docker images (#3381)
The kind clusters booted by the integration tests each had to pull
Prometheus and proxy-init images from the internet during linkerd
install.

Preemptively pull the images from the internet once, then execute `kind
load` commands for each of the clusters prior to starting integration
tests.

Depends on #3397

Signed-off-by: Andrew Seigner <siggy@buoyant.io>
2019-09-06 16:50:57 -07:00
Andrew Seigner 3be2b41a79
Replace Travis with GitHub Actions for master/tags (#3398)
GitHub Actions has been running unit and integration tests, in parallel
with Travis running those same tests, and also handling master merges
and tags.

This change completes the transtion to GitHub Actions, removing all
references to Travis. Similar to Travis, GitHub Actions now acts on
master merges and tag pushes by pushing Docker images to gcr.io, and
running integration tests against a GKE cluster.

Signed-off-by: Andrew Seigner <siggy@buoyant.io>
2019-09-06 15:58:46 -07:00
Andrew Seigner a8481b721a
GitHub Actions, kind, integration test logs fixes (#3372)
PR #3339 introduced a GitHub Actions CI workflow. Booting 6 clusters
simultaneously (3x Github Actions + 3x Travis) exhibits some transient
failures.

Implement fixes in GitHub Actions and integration tests to address kind
cluster creation and testing:
- Retry kind cluster creation once.
- Retry log reading from integration k8s clusters once.
- Add kind cluster creation debug logging.
- Add a GitHub Actions status badge to top of `README.md`.

Signed-off-by: Andrew Seigner <siggy@buoyant.io>
2019-09-04 12:44:27 -07:00
Andrew Seigner bfa251dd8b
Disable integration tests for forked PRs (#3377)
GitHub Action secrets are intentionally not available to forked PRs.
This causes the integration tests that require those secrets to fail.

Modify GitHub Actions such that they only run for non-forked PRs.

Signed-off-by: Andrew Seigner <siggy@buoyant.io>
2019-09-04 12:41:26 -07:00
Andrew Seigner 4f71b522dc
Introduce Github Actions CI workflow (#3339)
The existing Travis CI setup requires additional integrations and
permissions with Github, and also lacks some flexibility around job
dependency management.

Introduce a new CI workflow based on Github Actions. This initial
workflow performs the same CI work that Travis does, and will iniitially
run in parallel:
- Go unit tests
- JS unit tests
- Go lint
- Validate Go deps
- Integration tests (deep, upgrade, helm)

Signed-off-by: Andrew Seigner <siggy@buoyant.io>
2019-09-03 17:11:30 -07:00