The `proxy` and `web` Docker images were 161MB and 186MB, respectively.
Most of the space was tools installed into the `linkerd.io/base` image.
Decrease `proxy` and `web` Docker images to 73MB and 90MB, respectively.
Switch these images to be based off of `debian:stretch-20190812-slim`.
Also set `-ldflags "-s -w"` for `proxy-identity` and `web`. Modify
`linkerd.io/base` to also be based off of
`debian:stretch-20190812-slim`, update tag to `2019-09-04.01`.
Fixes#3383
Signed-off-by: Andrew Seigner <siggy@buoyant.io>
* Fix auto-injecting pods and integration tests reporting
When creating an Event when auto-injection occurs (#3316) we try to
fetch the parent object to associate the event to it. If the parent
doesn't exist (like in the case of stand-alone pods) the event isn't
created. I had missed dealing with one part where that parent was
expected.
This also adds a new integration test that I verified fails before this
fix.
Finally, I removed from `_test-run.sh` some `|| exit_code=$?` that was
preventing the whole suite to report failure whenever one of the tests
in `/tests` failed.
Signed-off-by: Alejandro Pedraza <alejandro@buoyant.io>
The integration tests under `/test` were run separately via l5d-bot,
lacking the feedback and job management provided by ci.
Enable integration tests in ci, via a docker build and kind clusters
executed on a remote DOCKER_HOST.
CI runs are now broken into two stages, run serially. Each stage is
composed of jobs run in parallel:
- Setup stage
- Validate go deps
- Remote docker build
- Kind cluster setup (deep)
- Kind cluster setup (upgrade)
- Kind cluster setup (helm)
- Test stage
- Go unit tests
- Node.js unit tests
- Kind integration tests (deep)
- Kind integration tests (upgrade)
- Kind integration tests (helm)
This PR also modifies `bin/test-run.sh` to always set `--failfast` for
Go tests.
Also introduce `bin/docker` and `bin/kubectl` scripts, to ensure
cacheable, pinned executables in ci.
The existing integration tests for master merges and docker pushes,
running against GKE, remain in place.
Signed-off-by: Andrew Seigner <siggy@buoyant.io>
The `bin/test-run` script executed upgrade, helm, and deep integration
test in series, but was structured in a way that did not permit running
these tests individually.
Move most of the logic from `bin/test-run` to a supporting library,
`bin/test-run.sh`, which will provide the ability to execute integration
tests individually. `bin/test-run`'s behavior is unchanged, it continues
to run upgrade, helm, and deep integration tests in series.
Signed-off-by: Andrew Seigner <siggy@buoyant.io>
`bin/helm` and `bin/protoc` were downloading their binaries into
`./target`, while `bin/lint` was downloading to the root of the repo.
Also travis was caching `./target`, which could become problematic if
that part of the test script relied on `target/cli/linux/linkerd`.
Standardize helm, kind, lint, and protoc to all download into
`./target/bin`, and modify travis to strictly cache that subdirectory.
Signed-off-by: Andrew Seigner <siggy@buoyant.io>
* Wait for `helm delete` to finish in integration test
Followup to #3251
In `helm_cleanup` block till the linkerd namespace has been deleted
Signed-off-by: Alejandro Pedraza <alejandro@buoyant.io>
PR #3247 introduced additional helm cleanup in `bin/test-cleanup`.
During the integration tests, `bin/test-cleanup` is called prior to
`helm_cleanup` in `bin/test-run`. This causes `helm_cleanup` to fail, as
resources have already been deleted by `bin/test-cleanup`, and the
integration tests fail with `FAIL: error cleaning up Helm`.
Modify the integration tests to first call `helm_cleanup` prior to
calling `bin/test-cleanup`.
Signed-off-by: Andrew Seigner <siggy@buoyant.io>
When helm integration tests fail, `bin/test-run` exits prior to calling
`helm_cleanup`, leaving behind a helm namespace and clusterrolebinding.
Update `bin/test-cleanup` to delete any remaining helm resources.
Signed-off-by: Andrew Seigner <siggy@buoyant.io>
* Delete symlink to old Helm chart
* Update 'install' code to use common Helm template structs
* Remove obsolete TLS assets functions.
These are now handle by Helm functions inside the templates
* Read defaults from values.yaml and values-ha.yaml
* Ensure that webhooks TLS assets are retained during upgrade
* Fix a few bugs in the Helm templates (see bullet points):
* Merge the way the 'install' ha and non-ha options are handled into one function
* Honor the 'NoInitContainer' option in the components templates
* Control plane mTLS will not be disabled if identity context in the
config map is empty. The data plane mTLS will still be automatically disabled
if the context is nil.
* Resolve test failures from rebase with master
* Fix linter issues
* Set service account mount path read-only field
* Add TLS variables of the webhooks and tap to values.yaml
During upgrade, these secrets are preserved to ensure they remain synced
wih the CA bundle in the webhook configurations. These Helm variables are used
to override the defaults in the templates.
* Remove obsolete 'chart' folder
* Fix bugs in templates
* Handle missing webhooks and tap TLS assets during upgrade
When upgrading from an older version that don't have these secrets, fallback to let Helm
create them by creating an empty charts.TLS struct.
* Revert the selector labels of webhooks to be compatible with that in 2.4
In 2.4, the proxy injector and profile validator webhooks already have their selector labels defined.
Since these attributes are immutable, the recent change to these selectors introduced by the Helm chart work will cause upgrade to fail.
* Alejandro's feedback
* Siggy's feedback
* Removed redundant unexported custom types
Signed-off-by: Ivan Sim <ivan@buoyant.io>
* Refactor proxy injection to use Helm charts
Fixes#3128
A new chart `/charts/patch` was created, that generates the JSON patch
payload that is to be returned to the k8s API when doing the injection
through the proxy injector, and it's also leveraged by the `linkerd
inject --manual` CLI.
The VFS was used by `linkerd install` to access the old chart under
`/chart`. Now the proxy injection also uses the Helm charts to generate
the JSON patch (see above) so we've moved the VFS from `cli/static` to a
new common place under `/pkg/charts/static`, and the new root for the VFS is
now `/charts`.
`linkerd install` hasn't yet migrated to use the new charts (that'll
happen in #3127), so the only change in that regard was the creation of
`/charts/chart` which is a symlink pointing to `/chart` that
`install.go` now uses, so that the VFS contains both the old and new
charts, as a temporary measure.
You can see that `/bin/Dockerfile-bin`, `/controller/Dockerfile` and
`/bin/build-cli-bin` do now `go generate` pointing to the new location
(and the `go generate` annotation was moved from `/cli/main.go` to
`pkg/charts/static/templates.go`).
The symlink trick doesn't work when building the binaries through
Docker, so `/bin/Dockerfile-bin` replaces the symlink with an actual
copy of `/chart`.
Also note that in `/controller/Dockerfile` we now need to include the
`prod` tag in `go install` like we do in `/bin/Dockerfile-bin` so that
the proxy injector does use the VFS instead of the local file system.
- The common logic to parse a chart has been moved from `install.go` to
`/pkg/charts/util.go`.
- The special ENV var in the proxy for "outbound router capacity" that
only applies to the Prometheus pod is now handled directly in the proxy
partial and all the associated go code could be removed.
- The `patch.go` lib for generating the JSON patch in go along
with its tests `patch_test.go` are no longer needed.
- Lots of functions in `/pkg/inject/inject.go` got removed/simplified
with their logic being moved into the charts themselves. As a
consequence lots of things in `inject_test.go` became irrelevant.
- Moved `template-values.go` from `/pkg/inject` to `pkg/charts` as that
contains the go structs representation of the chart variables that
will be leveraged in #3127.
Don't forget to run `/bin/helm.sh` whenever you make changes to charts
;-)
Signed-off-by: Alejandro Pedraza <alejandro@buoyant.io>
`bin/helm.sh`: you should see the following only if you have Tiller
installed in your cluster (which is installed with `helm init`):
```
Performing dry run installation
NAME: linkerd
Performing dry run installation (HA mode)
NAME: linkerd
```
`bin/test-cleanup`: when linkerd is not installed:
Before:
```bash
$ bin/test-cleanup
cleaning up control-plane namespaces in k8s-context []
cleaning up data-plane namespaces in k8s-context []
cleaning up rolebindings in kube-system namespace in k8s-context []
```
After this PR's changes:
```
$bin/test-cleanup
cleaning up control-plane namespaces in k8s-context []
no control-plane namespaces found
cleaning up data-plane namespaces in k8s-context []
no data-plane namespaces found
no clusterrolebindings found
no clusterroles found
no mutatingwebhookconfigurations found
no validatingwebhookconfigurations found
no podsecuritypolicies found
no customresourcedefinitions found
no apiservices found
cleaning up rolebindings in kube-system namespace in k8s-context []
```
Signed-off-by: Alejandro Pedraza <alejandro@buoyant.io>
* Updated controller template with proxy partials
* Declare dependency in requirements.yaml
* Add partial template for proxy's metadata
* Add proxy-init partial template
* Script to lint Helm charts and update their dependencies
* Update partials chart Chart.yaml
* Add proxy-init and resource partial templates
* Replace hard coded namespace variable in proxy env var
* Ignore chart dependencies .tgz files
* Add missing fields and re-order YAML elements to match CLI output
* Reuse control plane's resource partial template in 'partials' chart
* Set the proxy's destination service address env var
* Add Grafana's template
* Update api version of controller RBAC
* Add Heartbeat template
* Remove duplicated resources partial template
* Add remainder control plane components templates
* Add template for the 'linkerd-config' config map
* Add debug container template
* Update proxy partial with 'disable-identity' and 'disable-tap' variables
Note that these are inject-only variables.
Also added the LINKERD2_PROXY_TAP_SVC_NAME env var.
* Add validation conditions to ensure identity and tap aren't disabled for
control plane components
* Add partials for service account token mount path and security context capabilities
* Change proxy and proxy-init templates to use global scope
Some of the nested variables are removed from values.yaml to ensure changes
made to root-level variables are propagated directly into the partial
templates. The previous approach of using YAML anchors in the
values.yaml to share common values can get out-of-sync when values are
changed via the Helm's `--set` option.
* Update templates and values file to match #3161
* Perform a dry run installation if there is a local Tiller
* Reorder JSON elements in linkerd-config
* Re-adjust nested partials indentation to work with inject 'patch' chart
Previously, the partials will render their content as an element in the list.
While it works for installation, the toJson function in the 'inject' patch code
ends up converting it into a JSON list, instead of the expected JSON
object.
* Trap the last fail command in the Helm shell script
* Add the identity trust anchor
* Address Thomas' feedback on handling HA
All the HA-related variables are moved to values-ha.yaml
* Convert ignore ports string to JSON list in linkerd-config
Also fixed some indentation issues.
* Add values-ha.yaml
* Include the service account token mount path only if identity is enabled
* Fixed malformed JSON in linkerd-config config map
* Rename chart to 'linkerd2'
* Add NOTES.txt
* Fix incorrect variable path in proxy template
* Remove fake TLS assets
* Add 'required' constraint to identity trust anchors variable
* Update tap templates per #3167
* Bump default version to edge-19.8.1 due to dependency on RSA support
Signed-off-by: Ivan Sim <ivan@buoyant.io>
The Tap Service enabled tapping of any meshed pod, regardless of user
privilege.
This change introduces a new Tap APIService. Kubernetes provides
authentication and authorization of Tap requests, and then forwards
requests to a new Tap APIServer, which implements a Kubernetes
aggregated APIServer. The Tap APIServer authenticates the client TLS
from Kubernetes, and authorizes the user via a SubjectAccessReview.
This change also modifies the `linkerd tap` command to make requests
against the new APIService.
The Tap APIService implements these Kubernetes-style endpoints:
POST /apis/tap.linkerd.io/v1alpha1/watch/namespaces/:ns/tap
POST /apis/tap.linkerd.io/v1alpha1/watch/namespaces/:ns/:res/:name/tap
GET /apis
GET /apis/tap.linkerd.io
GET /apis/tap.linkerd.io/v1alpha1
GET /healthz
GET /healthz/log
GET /healthz/ping
GET /metrics
GET /openapi/v2
GET /version
Users authorize to the new `tap.linkerd.io/v1alpha1` via RBAC. Only the
`watch` verb is supported. Access is also available via subresources
such as `deployments/tap` and `pods/tap`.
This change introduces the following resources into the default Linkerd
install:
- Global
- APIService/v1alpha1.tap.linkerd.io
- ClusterRoleBinding/linkerd-linkerd-tap-auth-delegator
- `linkerd` namespace:
- Secret/linkerd-tap-tls
- `kube-system` namespace:
- RoleBinding/linkerd-linkerd-tap-auth-reader
Tasks not covered by this PR:
- `linkerd top`
- `linkerd dashboard`
- `linkerd profile --tap`
- removal of the unauthenticated tap controller
Fixes#2725, #3162, #3172
Signed-off-by: Andrew Seigner <siggy@buoyant.io>
The repo relied on `dep` for managing Go dependencies. Go 1.11 shipped
with Go modules support. Go 1.13 will be released in August 2019 with
module support enabled by default, deprecating GOPATH.
This change replaces `dep` with Go modules for dependency management.
All scripts, including Docker builds and ci, should work without any dev
environment changes.
To execute `go` commands directly during development, do one of the
following:
1. clone this repo outside of `GOPATH`; or
2. run `export GO111MODULE=on`
Summary of changes:
- Docker build scripts and ci set `-mod=readonly`, to ensure
dependencies defined in `go.mod` are exactly what is used for the
builds.
- Dependency updates to `go.mod` are accomplished by running
`go build` and `go test` directly.
- `bin/go-run`, `bin/build-cli-bin`, and `bin/test-run` set
`GO111MODULE=on`, permitting usage inside and outside of GOPATH.
- `gcr.io/linkerd-io/go-deps` tags hashed from `go.mod`.
- `bin/update-codegen.sh` still requires running from GOPATH,
instructions added to BUILD.md.
Fixes#1488
Signed-off-by: Andrew Seigner <siggy@buoyant.io>
The openAPIV3Schema validation in the ServiceProfiles CRD is very limited in what it can validate and is obviated by more sophisticated validation done by the validating admission controller. Therefore, we would like to remove the openAPIV3Schema validation to reduce the size and complexity of the CRD object.
To do so, we must also bump the version of the ServiceProfile custom resource from v1alpha1 to v1alpha2. This ensures that when the controller is upgraded, it will attempt to watch the v1alpha2 resource. If it cannot (because, for example, the controller pod started before the ServiceProfile CRD was updated and therefore the v1alpha2 version does not exist) then it will go into a crash loop backoff until it can. This essentially means that the controller will wait for the CRD to be upgraded to include v1alpha2 before it will start.
Bumping the version is necessary because if we did not, it would be possible for the controller to start before the CRD is updated (removing the validation). In this case, when the CRD is edited, the controller will lose its list watch on ServiceProfiles and will stop getting updates.
Signed-off-by: Alex Leong <alex@buoyant.io>
* Updates for the integration test script
1. Remove existing resources prior to starting the test
2. Remove existing resources post upgrade test
3. Fail fast if 'install_test.go` fails
4. Don't perform cleanup if any of the tests fail for debugging
opportunity
* Remove pre-test cleanup from .travis.yaml
This is now done in the bin/test-run script so that it can be shared
between l5d-bot and staging.
Signed-off-by: Ivan Sim <ivan@buoyant.io>
Integration tests may fail and leave behind namespaces that following
builds aren't able to clean up because the git sha is being included in
the namespace name, and the following builds don't know about those
shas.
This modifies the `test-cleanup` script to delete based on object labels
instead of relying on the objects names, now that after 2.4 all the
control plane components are labeled. Note that this will also remove
non-testing linkerd namespaces, but we were already kinda doing that
partially because we were removing the cluster-level resources (CRDs,
webhook configs, clusterroles, clusterrolebindings, psp).
`test-cleanup` no longer receives a namespace name as an argument.
The data plane namespaces aren't labeled though, so I've added the
`linkerd.io/is-test-data-plane` label for them in
`CreateNamespaceIfNotExists()`, and making sure all tests that need a
data plaine explicitly call that method instead of creating the
namespace as a side-effect in `KubectlApply()`.
Signed-off-by: Alejandro Pedraza <alejandro@buoyant.io>
The git-commt-proxy-version script attempted to link to the specifc
SHAs. GitHub doesn't actually render these links, and the information is
redundant since we link the appropriate PR.
Each time we update the proxy from the linkerd2-proxy repo, we make the
change slightly differently. The bin/git-commit-proxy-version does all the
steps needed to update the proxy version up to and including making a
commit to this repo.
The proxy version is now stored in a .proxy-version file and is
consumed directly by Dockerfile-proxy, which both simplifies the
Dockerfile and the update process.
This script formats commit messages and emits output as follows:
```
commit c05198a851f69bdc7007974a0ef1f4c01c98d0ce (HEAD -> ver/proxy-update)
Author: Oliver Gould <ver@buoyant.io>
Date: Thu Jul 11 17:23:05 2019 +0000
proxy: Update to linkerd/linkerd2-proxy#3a3ec3b
* linkerd/linkerd2-proxy#0cc58cd fallback: Clarify fallback layering (linkerd/linkerd2-proxy#288)
* linkerd/linkerd2-proxy#b71349a Replace `log` and `env-logger` with `tracing` and `tracing-fmt` (linkerd/linkerd2-proxy#277)
* linkerd/linkerd2-proxy#3a3ec3b Use a constant-time load balancer (linkerd/linkerd2-proxy#266)
diff --git a/.proxy-version b/.proxy-version
index f81f40de..d7faa12d 100644
--- a/.proxy-version
+++ b/.proxy-version
@@ -1 +1 @@
-05b012d
+3a3ec3b
```
Integration tests on master broke following the 2.4 release, caused by
the recent disabling of multi control-plane support, coupled with the
upgrade integration test (which now upgrades from 2.4 to current sha).
The integration tests do the following:
1. install the current sha
2. test the current sha
3. install the latest stable in an `upgrade` namespace
4. in the `upgrade` namespace, upgrade from stable to latest sha
5. test the upgraded installation
Step 3 breaks because `linkerd install` with stable-2.4 will fail if
existing global resources (from step 1) are present.
For now, modify the integration tests to do the following:
1. install the latest stable in an `upgrade` namespace
2. in the `upgrade` namespace, upgrade from stable to latest sha
3. test the upgraded installation
4. upon successful step 3, remove all related resources
5. install the current sha
6. test the current sha
Signed-off-by: Andrew Seigner <siggy@buoyant.io>
Pick up the following proxy changes:
* Update httparse to v1.3.4
* canonicalize: stop resolving when the receiver is dropped
* router: Remove interval from router eviction
Signed-off-by: Alex Leong <alex@buoyant.io>
* Introduce new checks to determine existence of global resources and the
'linkerd-config' config map.
* Update pre-check to check for existence of global resources
This ensures that multiple control planes can't be installed into
different namespaces.
* Update integration test clean-up script to delete psp and crd
Signed-off-by: Ivan Sim <ivan@buoyant.io>
* Fixed the proxy rejecting HTTP2 requests that don't have an `:authority`
* Improved idle service eviction to reduce resource consumption for clients
that send requests to many services
Signed-off-by: Kevin Leimkuhler <kleimkuhler@icloud.com>
commit 790a86aa9db463af479647bb91b8b55280d74d4
Author: Sean McArthur <sean@buoyant.io>
Date: Tue Jun 4 20:28:05 2019 -0700
Update h2 to v0.1.23 (#264)
- Fixes leaked DATA frames if never polled.
Signed-off-by: Sean McArthur <sean@buoyant.io>
Split proxy-init into separate repo
Fixes#2563
The new repo is https://github.com/linkerd/linkerd2-proxy-init, and I
tagged the latest there `v1.0.0`.
Here, I've removed the `/proxy-init` dir and pinned the injected
proxy-init version to `v1.0.0` in the injector code and tests.
`/cni-plugin` depends on proxy-init, so I updated the import paths
there, and could verify CNI is still working (there is some flakiness
but unrelated to this PR).
For consistency, I added a `--init-image-version` flag to `linkerd
inject` along with its corresponding override config annotation.
Signed-off-by: Alejandro Pedraza <alejandro@buoyant.io>
In #2679 we introduced an upgrade integration test. At the time we only
supported upgrading from a recent edge. Since that PR, a stable build
was released supporting upgrade.
Modify the upgrade integration test to upgrade from the latest stable
rather than latest edge. This fulfills the original intent of #2669.
Also add some known k8s event warnings.
Signed-off-by: Andrew Seigner <siggy@buoyant.io>
commit b27dfb2d21aa8ca5466ea0edce17d27094ace7c1
Author: Takanori Ishibashi <takanori.1112@gmail.com>
Date: Wed May 15 05:58:42 2019 +0900
updaes->updates (#250)
Signed-off-by: Takanori Ishibashi <takanori.1112@gmail.com>
commit 16441c25a9d423a6ab12b689b830d9ae3798fa00
Author: Eliza Weisman <eliza@buoyant.io>
Date: Tue May 14 14:40:03 2019 -0700
Pass router::Config directly to router::Layer (#253)
Currently, router `layer`s are constructed with a single argument, a
type implementing `Recognize`. Then, the entire router stack is built
with a `router::Config`. However, in #248, it became necessary to
provide the config up front when constructing the `router::layer`, as
the layer is used in a fallback layer. Rather than providing a separate
type for a preconfigured layer, @olix0r suggested we simply change all
router layers to accept the `Config` when they're constructed (see
https://github.com/linkerd/linkerd2-proxy/pull/248#discussion_r283575008).
This branch changes `router::Layer` to accept the config up front. The
`router::Stack` types `make` function now requires no arguments, and the
implementation of `Service` for `Stack` can be called with any `T` (as
the target is now ignored).
Signed-off-by: Eliza Weisman <eliza@buoyant.io>
commit b70c68d4504a362eac6a7828039a2e5c7fcd308a
Author: Eliza Weisman <eliza@buoyant.io>
Date: Wed May 15 13:14:04 2019 -0700
Load balancers fall back to ORIG_DST when no endpoints exist (#248)
Currently, when no endpoints exist in the load balancer for a
destination, we fail the request. This is because we expect endpoints to
be discovered by both destination service queries _and_ DNS lookups, so
if there are no endpoints for a destination, it is assumed to not exist.
In linkerd/linkerd2#2661, we intend to remove the DNS lookup from the
proxy and instead fall back to routing requests for which no endpoints
exist in the destination service to their SO_ORIGINAL_DST IP address.
This means that the current approach of failing requests when the load
balancer has no endpoints will no longer work.
This branch introduces a generic `fallback` layer, which composes a
primary and secondary service builder into a new layer. The primary
service can fail requests with an error type that propages the original
request, allowing the fallback middleware to call the fallback service
with the same request. Other errors returned by the primary service are
still propagated upstream.
In contrast to the approach used in #240, this fallback middleware is
generic and not tied directly to a load balancer or a router, and can
be used for other purposes in the future. It relies on the router cache
eviction added in #247 to drain the router when it is not being used,
rather than proactively destroying the router when endpoints are
available for the lb, and re-creating it when they exist again.
A new trait, `HasEndpointStatus`, is added in order to allow the
discovery lookup to communicate the "no endpoints" state to the
balancer. In addition, we add a new `Update::NoEndpoints` variant to
`proxy::resolve::Update`, so that when the control plane sends a no
endpoints update, we switch from the balancer to the no endpoints state
_immediately_, rather than waiting for all the endpoints to be
individually removed. When the balancer has no endpoints, it fails all
requests with a fallback error, so that the fallback middleware
A subsequent PR (#248) will remove the DNS lookups from the discovery
module.
Closes#240.
Signed-off-by: Eliza Weisman <eliza@buoyant.io>
commit 6525b0638ad18e74510f3156269e0613f237e2f5
Author: Zahari Dichev <zaharidichev@gmail.com>
Date: Wed May 15 23:35:09 2019 +0300
Allow disabling tap by setting an env var (#252)
This PR fixeslinkerd/linkerd2#2811. Now if
`LINKERD2_PROXY_TAP_DISABLED` is set, the tap is not served at all. The
approach taken is that the `ProxyParts` is changed so the
`control_listener` is now an `Option` that will be None if tap is
disabled as this control_listener seems to be exclusively used to serve
the tap. Feel free to suggest a better approach.
Signed-off-by: Zahari Dichev <zaharidichev@gmail.com>
commit 91f32db2ea6d74470fd689c713ff87dc7586222d
Author: Zahari Dichev <zaharidichev@gmail.com>
Date: Thu May 16 00:45:23 2019 +0300
Assert that outbound TLS works before identity is certified (#251)
This commit introduces TLS capabilities to the support server as well as
tests to ensure that outbound TLS works even when there is no verified
certificate for the proxy yet.
Fixeslinkerd/linkerd2#2599
Signed-off-by: Zahari Dichev <zaharidichev@gmail.com>
commit 45aadc6b1b28e6daea0c40e694a86ae518887d85
Author: Sean McArthur <sean@buoyant.io>
Date: Wed May 15 14:25:39 2019 -0700
Update h2 to v0.1.19
Includes a couple HPACK fixes
Signed-off-by: Sean McArthur <sean@buoyant.io>
commit 3e0e00c6dfbf5a9155b887cfd594f611edfc135f
Author: Oliver Gould <ver@buoyant.io>
Date: Thu May 16 08:11:06 2019 -0700
Update mio to 0.6.17 (#257)
To pick up https://github.com/tokio-rs/mio/pull/939
commit 5f89351081eff47a4ab8cd88e2e1a69a04f86541
Author: Oliver Gould <ver@buoyant.io>
Date: Thu May 9 16:39:24 2019 -0700
Upgrade tower dependencies (#249)
Tower must be updated in order to pickup tower-rs/tower#281
to address linkerd/linkerd2#2804.
This adopts released crates where possible.
commit 5d5eed6f8180b8db4090d995e71fdf7b0890c647
Author: Zahari Dichev <zaharidichev@gmail.com>
Date: Thu May 9 01:08:34 2019 +0300
Assert that TLS connection is refused if identity is not certified yet (#243)
This branch adds tls capability to the support cient used in tests. In addition to that it adds two tests verifying that a TLS connection is refused in case the identity is not certified yet. This attempts to fix #https://github.com/linkerd/linkerd2/issues/2598 and provide facility to write tests for https://github.com/linkerd/linkerd2/issues/2676.
As these are still some of my first lines of Rust code, it is advised to approach everything with a healthy dose of doubt :)
Signed-off-by: Zahari Dichev <zaharidichev@gmail.com>
commit 1b9bb3745e44c959d1d41d14fed2b2822c82b5ba
Author: Oliver Gould <ver@buoyant.io>
Date: Wed May 8 14:28:37 2019 -0700
Introduce dispatch timeouts around buffers (#246)
The proxy has several buffers, especially where it routes requests over
shared stacks. If any of these routes is unavailable, then a request may
remain buffered indefinitely. Previously, before service profiles were
introduced, there was a default _response_ timeout that would cause
these requests to fail; but since this response timeout is now optional
(and is only applied once the request has been routed within a proxy),
then we need a new mechanism to prevent requests from getting "stuck".
This change does the following:
- all proxied requests are annotated with a dispatch deadline;
- each time a request is bufered, a timeout is registered.
- if the timeout fires, the response exception fails, a 503 is returned,
and the request is dropped.
- if the request is processed into the inner stack, the timeout is
ignored.
The dispatch timeout limits the _time a request is buffered in a proxy_.
This is distinct from the response timeout, as the server's response may
naturally be delayed for any number of (non-proxy-related) reasons.
The `insert_target` module has been generalized to `insert` to support
setting the DispatchDeadline extension.
The `buffer` module has been augmented with generic deadline-extraction
logic.
The `svc` module now exposes its own builder type that notably adds
a `buffer_pending` helper. It's helpful to pull a builder type into the
proxy to assist debugging type errors when modifying stacks.
Fixeslinkerd/linkerd2#2779linkerd/linkerd2#2795
commit caf899557c3b041190f63544da865396231b3e30
Author: Oliver Gould <ver@buoyant.io>
Date: Fri May 3 15:55:32 2019 -0700
router: Fail requests when the route is not ready (#241)
In linkerd/linkerd2#2779, we plan to expire requests while they are
buffered. However, the router _implicitly_ buffers requests in the
executor when the inner service is not ready.
This change alters the route to wrap all inner layers in a `LoadShed`
so it can expect all services to `poll_ready()` immediately.
commit 587bad101d9e5daeacb24b6733097c350a798356
Author: Eliza Weisman <eliza@buoyant.io>
Date: Fri May 3 14:18:08 2019 -0700
Remove Destination service query concurrency limit (#244)
Currently, the proxy enforces a limit on the number of concurrent
queries (i.e., the number of gRPC streams) to the Destination service.
This limit was added based on information about the behaviour of the
Destination service that is now known to be incorrect.
This branch removes the limit on concurrent queries from the proxy's
`control::destination` module. Although it should now be possible to
simplify this code as a result of this change, I've refrained from doing
any major refactoring in this branch --- my intention is to do this
after the DNS fallback behaviour has also been removed, as together with
this change, that will result in a _significant_ simplification of the
module. Additionally, I've removed the tests for the concurrency limit,
as they are no longer relevant.
The `LINKERD2_PROXY_DESTINATION_CLIENT_CONCURRENCY_LIMIT`
environment variable was also removed; this is not a breaking change as
neither the CLI nor the proxy injector will currently set this env var.
Signed-off-by: Eliza Weisman <eliza@buoyant.io>
commit cbdf45b44f7e4d852dc0497716062167ab9539fb
Author: Sean McArthur <sean@buoyant.io>
Date: Thu May 2 11:47:48 2019 -0700
Remove h2::Error requirement from metrics
Signed-off-by: Sean McArthur <sean@buoyant.io>
commit 3276949d4608dc4344b7bed3de2fc4b3080c2c6e
Author: Sean McArthur <sean@buoyant.io>
Date: Thu May 2 09:44:00 2019 -0700
delete unused proxy::http::metrics::class module
Signed-off-by: Sean McArthur <sean@buoyant.io>
Signed-off-by: Eliza Weisman <eliza@buoyant.io>
The `docker-build-proxy` script builds `Dockerfile-proxy`. That
Dockerfile depends on a go-deps image, and takes a `LINKERD_VERSION`
arg. The `docker-build-proxy` script was neither ensuring go-deps had
been built, nor setting `LINKERD_VERSION`. The former resulted in the
build failing if go-deps did not exist. The latter resulted in
`dev-undefined` log messages in the `linkerd-proxy` container.
Fix `docker-build-proxy` to ensure go-deps are built, and also set the
`LINKERD_VERSION`. This brings this script more in-line with the other
`docker-build-*` scripts.
Signed-off-by: Andrew Seigner <siggy@buoyant.io>
commit 073a1beb4a7cd709c6b1eaa56a319c1829a94d11
Author: Sean McArthur <sean@buoyant.io>
Date: Mon Apr 29 17:54:01 2019 -0700
tap: remove need to clone Services (#238)
This refactors the tap system to not require intermediary channels to
register matches and taps when a request comes through. The Dispatcher
that used to exist in order to prevent tapping more requests than the
limit asked for has been removed. In its place is a shared atomic
counter to keep the count under the limit.
The resulting behavior should be the same. There should be improved
performance as tap registration doesn't need go through a second
channel, and requests don't need to be delayed waiting for the
dispatcher to be able to process its queue.
Signed-off-by: Sean McArthur <sean@buoyant.io>
commit 7a3be8c8737188e5debbc465f9a33da0d79b8b80
Author: Zahari Dichev <zaharidichev@gmail.com>
Date: Wed May 1 01:57:01 2019 +0300
Replace fixed reconnect backoff with exponential one (#237)
When reconnecting to a destination, use an exponential, jittered backoff strategy.
Signed-off-by: Zahari Dichev <zaharidichev@gmail.com>
commit 32b813aad4fe2fcf0252e8c2215d6835101d2337
Author: Oliver Gould <ver@buoyant.io>
Date: Tue Apr 30 15:58:20 2019 -0700
Support endpoint weights (#230)
This change modifies the proxy to honor weights provided by the
destination service. When the destination service replies with a
weight, this value is divided by 10,000 to produce a weight on
[0.0, ~400000.0]. This weight is used by load the load balancer
to modify load interpretation and therefore request distribution.
A weight of 0.0 will cause the endpoint's load to be effectively infinite
so that requests will only be sent to the endpoint when no other endpoints
exists or when the other endpoints that were considered had 0-weights.
commit 501802671a346250b6dbaae73f29d9be7a4c2086
Author: Sean McArthur <sean@buoyant.io>
Date: Wed May 1 13:42:38 2019 -0700
Remove buffers from endpoint stacks (#239)
Due to the `http::settings::router`, a `buffer` was needed in each
endpoint stack. This meant that the service was always ready, even if
the client were falling over (and reconnecting). In turn, this meant
that the balancer would pick one of these endpoint stacks, because it
was always ready!
This change includes a test of a failing endpoint, that the balancer no
longer assumes it is ready, and has the following functional changes:
- Removed `http::settings::router`, instead the client HTTP settings are
detected as part of the `DstAddr`. This means that each balancer only
has endpoints with the same HTTP settings.
- Removed `buffer` layer from inside the endpoint stacks.
Signed-off-by: Sean McArthur <sean@buoyant.io>
When developing on the proxy, it's convenient to build the proxy while
the linkerd2 image is building at a given tag; but because the proxy is
built last, it's difficult to build the proxy at the same tag
simultaneously.
This is made easier by building the proxy first so that the parallel
build can be initiated after this. This shouldn't impact other
development workflows.
commit 61db2e77a247f7b0235b67581f60e8a92f8543cb
Author: Sean McArthur <sean@seanmonstar.com>
Date: Tue Apr 23 17:20:43 2019 -0700
Replace linkerd2-stack with tower-layer (#236)
Signed-off-by: Sean McArthur <sean@buoyant.io>
commit 2d6c7145cadf709832f3507bcefdaee509ebde81
Author: Sean McArthur <sean@seanmonstar.com>
Date: Thu Apr 18 12:40:48 2019 -0700
Add load shedding when over max-in-flight requests. (#225)
Also adds configuration for inbound and outbound max-in-flight requests.
Signed-off-by: Sean McArthur <sean@buoyant.io>
commit f4b5cd0b4a25d7d942e018b42af1157ae2e7dbb0
Author: Oliver Gould <ver@buoyant.io>
Date: Wed Apr 17 13:53:49 2019 -0700
Upgrade tower (#232)
This avails the proxy of newer load balancer features, an updated buffer
implementation, etc.
The new buffer implementation requires that we implement TypedExecutor
for our logging executor; and more error types have been made dynamic.
In https://github.com/linkerd/linkerd2-proxy/pull/233, we fixed an issue
in the proxy where, when the proxy performed TLS discovery (on inbound
connections), detection on a slow or idle connection could block all
other connections from being accepted on the listener.
Fixes#2581#2585#2630
* proxy: Bump pinned version to f2d907b
This change picks up:
* Added configuration for overriding the connection backoff
* Added configuration for overriding the HTTP/2 stream or connection window size
* Disable potentially info-leaking header
Signed-off-by: Kevin Leimkuhler <kevin@kleimkuhler.com>
Introduce a `bin/test-scale` script to deploy Linkerd alongside sample
apps at scale.
This script deploys the following:
- Linkerd control-plane, with service profiles
- 5 namespaces x 5 replicas of each:
- Emojivoto demo app
- Books demo app, with service profiles
- Lifecycle / bb test environment
Fixes#2517
Signed-off-by: Andrew Seigner <siggy@buoyant.io>
The integration tests were not exercising proxy auto inject.
Introduce a `--proxy-auto-inject` flag to `install_test.go`, which
now exercises install, check, and smoke test deploy for both manual and
auto injected use cases.
Part of #2569
Signed-off-by: Andrew Seigner <siggy@buoyant.io>
Introduce a `bin/test-clouds` and cleanup script, to run integration
tests against 4 cloud providers.
Also modify the integration tests to accept a `--context` param to
specify the Kubernetes context to run the tests against.
Fixes#2516
Signed-off-by: Andrew Seigner <siggy@buoyant.io>
Adds local and cloud integration testing for the dashboard using WebdriverIO and
SauceLabs. Includes documentation on how to set up and run the Sauce Connect
proxy locally. Adds a `bin/web integration` script that takes `local` or `cloud`
arguments to run the tests.
Note: for web development, the web server launched by `bin/web run` and `bin/web
dev` is now 7777, not 8084, because the Sauce Connect proxy can only tunnel to
certain ports.
Picks up the following proxy change:
* Add a oneshot to notify the profiles daemon if the stream is dropped
Signed-off-by: Alex Leong <alex@buoyant.io>
The new proxy has changed its configuration as follows:
- `LISTENER` urls are now `LISTEN_ADDR` addresses;
- `CONTROL_URL` is now `DESTINATION_SVC_ADDR`;
- `*_NAMESPACE` vars are no longer needed;
- The `PROXY_ID` is now the `DESTINATION_CONTEXT`;
- The "metrics" port is now the "admin" port, since it serves more than
just metrics;
- A readiness probe now checks a dedicated /ready endpoint eagerly.
Identity injection is **NOT** configured by this branch.
The proxy's TLS implementation has changed to use a new _Identity_ controller.
In preparation for this, the `--tls=optional` CLI flag has been removed
from install and inject; and the `ca` controller has been deleted. Metrics
and UI treatments for TLS have **not** been removed, as they will continue to
be valuable for the new Identity system.
With the removal of the old identity scheme, the Destination service's proxy
ID field is now set with an opaque string (e.g. `ns:emojivoto`) to enable
locality awareness.
`bin/test-cleanup` takes 48s on ci.
This change sets `kubectl --wait=false`, so the command should return
immediately rather than waiting for resources to be fully deleted.
Signed-off-by: Andrew Seigner <siggy@buoyant.io>
linkerd/linkerd2#2428 modified SelfSubjectAccessReview behavior to no
longer paper-over failed ServiceProfile checks, assuming that
ServiceProfiles will be required going forward. There was a lingering
ServiceProfile check in the web's startup that started failing due to
this change, as the web component does not have (and should not need)
ServiceProfile access. The check was originally implemented to inform
the web component whether to expect "single namespace" mode or
ServiceProfile support.
Modify the web's initialization to always expect ServiceProfile support.
Also remove single namespace integration test
Signed-off-by: Andrew Seigner <siggy@buoyant.io>
proxy: bump pinned version to 7e55196
This picks up the following commit:
* 7e55196 Bump tower-grpc (linkerd/linkerd2-proxy#202)
The new `tower-grpc` version (tower-rs/tower-grpc#115) improves the
messages attached to internal gRPC issues. This will aid significantly
in debugging the proxy's gRPC communication with the control plane.
This picks up the following commits:
* 0fe8063 replace `Error::cause` with `Error::source` (#2370) (linkerd/linkerd2-proxy#201)
* 1ea7559 Minor cleanup in the config tests (linkerd/linkerd2-proxy#188)
* d0ef56b Update *ring* to 0.14.6 (linkerd/linkerd2-proxy#197)
* c54377f fs-watch: Use a properly sized buffer for inotify events (linkerd/linkerd2-proxy#195)
* 23e02a6 Update Router to wait for inner poll_ready before calling inner call
* 2de8e9b Update metrics quickcheck to 0.8, and hyper to 0.12.24
* d1bbd4b make: Optionally include debug symbols with builds (linkerd/linkerd2-proxy#193)
* 738a541 Fix compilation warnings in fs-watch (linkerd/linkerd2-proxy#192)
* 6cc7558 Apply rustfmt (linkerd/linkerd2-proxy#191)
Signed-off-by: Ivan Sim <ivan@buoyant.io>
linkerd/linkerd#2349 introduced ServiceProfile CRD deletion to
`bin/test-cleanup`. Unfortunately that CRD is cluster-wide and shared
across any Linkerd's currently installed.
Revert CRD deletion.
Signed-off-by: Andrew Seigner <siggy@buoyant.io>
The control-plane components relied on a `--single-namespace` param,
passed from `linkerd install` into each individual component, to
determine which namespaces they were authorized to access, and whether
to support ServiceProfiles. This command-line flag was redundant given
the authorization rules encoded in the parent `linkerd install` output,
via [Cluster]Role[Binding]s.
Modify the control-plane components to query Kubernetes at startup to
determine which namespaces they are authorized to access, and whether
ServiceProfile support is available. This allows removal of the
`--single-namespace` flag on the components.
Also update `bin/test-cleanup` to cleanup the ServiceProfile CRD.
TODO:
- Remove `--single-namespace` flag on `linkerd install`, part of #2164
Signed-off-by: Andrew Seigner <siggy@buoyant.io>
Homebrew/homebrew-core#36957 introduces a brew formula for the linkerd
cli. It depends on `bin/build-cli-bin` to build a local linkerd cli
binary.
This change adds a note to `bin/build-cli-bin`, to consider brew when
making changes to that script.
Signed-off-by: Andrew Seigner <siggy@buoyant.io>
The `bin/test-cleanup` script was correctly deleting all namespaces
created by `bin/test-run`, but was leaving behind clusterroles and
clusterrolebindings, defined cluster-wide.
Update `test-cleanup` to delete clusterroles and clusterrolebindings
created by `test-run`.
Signed-off-by: Andrew Seigner <siggy@buoyant.io>
The `bin/build-cli-bin` script, intended to build a local `linkerd` cli
binary, was compiling the binary configured to read template files out
of the local machine's GOPATH.
This change modifies `build-cli-bin` to build a `linkerd` binary the
same way `docker-build-cli-bin` does. Specifically, by generating static
template files for inclusion in the build, and adding the `-tags prod`
flag to ensure those files are compiled in.
Signed-off-by: Andrew Seigner <siggy@buoyant.io>
Define the global and proxy configs protobuf types that will be used by CLI install, inject and the proxy-injector.
Signed-off-by: Ivan Sim <ivan@buoyant.io>
golangci-lint disables some checks for golint, including checks for
well-formed comments on all exported symbols
This change disables the golangci-lint's `exclude-use-default` setting,
to run golint with default settings.
Also introduce a `.golangci.yml` file to centralize config.
Signed-off-by: Andrew Seigner <siggy@buoyant.io>
`golangci-lint` performs numerous checks on Go code, including golint,
ineffassign, govet, and gofmt.
This change modifies `bin/lint` to use `golangci-lint`, and replaces
usage of golint and govet.
Also perform a one-time gofmt cleanup:
- `gofmt -s -w controller/`
- `gofmt -s -w pkg/`
Part of #217
Signed-off-by: Andrew Seigner <siggy@buoyant.io>
commit 59d00f69653730353ec246b8cb2eb39d80a54d3e
Author: Oliver Gould <ver@buoyant.io>
Date: Mon Feb 11 10:51:37 2019 -0800
Log canonicalization warnings on only the first error (#189)
When a canonicalization task fails to resolve a name, our logging is not
particularly clear about the current state of the stack. Specifically,
it's difficult to know whether the stack has resolved the name
successfully before.
With this change, canonicalization failures are logged (at warning, not
error) only when the task has not previously resolved a name.
Subsequent errors are now logged at the debug level (instead of
warning).
The Proxy API service lacked introspection of its internal state.
Introduce a new gRPC Discovery API, implemented by two servers:
1) Proxy API Server: returns a snapshot of discovery state
2) Public API Server: pass-through to the Proxy API Server
Also wire up a new `linkerd endpoints` command.
Fixes#2165
Signed-off-by: Andrew Seigner <siggy@buoyant.io>
* Remove destination address from endpoint metric labels
(linkerd/linkerd2#187)
* Set proxy_id in calls to Get and GetProfile (linkerd/linkerd2#183)
* Add l5d-client-id on inbound requests if meshed TLS (linkerd/linkerd2#184)
Signed-off-by: Kevin Leimkuhler <kevin@kleimkuhler.com>
In linkerd/linkerd2-proxy#186, the proxy supports configuration of TCP
keepalive values.
This change sets `LINKERD2_PROXY_INBOUND_ACCEPT_KEEPALIVE` and
`LINKERD2_PROXY_OUTBOUND_CONNECT_KEEPALIVE` to 10s when injecting the
proxy, so that remote connections are configured with a keepalive.
This configuration is NOT yet exposed through the CLI. This may be done
in a followup, if necessary.
Fixes#1949
* Export RootOptions and BuildFirewallConfiguration so that the cni-plugin can use them.
* Created the cni-plugin based on istio-cni implementation
* Create skeleton files that need to be filled out.
* Create the install scripts and finish up plugin to write iptables
* Added in an integration test around the install_cni.sh and updated the script to handle the case where it isn't the only plugin. Removed the istio kubernetes.go file in favor of pkg/k8s; initial usage of this package; found and fixed the typo in the ClusterRole and ClusterRoleBinding; found the docker-build-cni-plugin script
* Corrected an incorrect name in the docker build file for cni-plugin
* Rename linkerd2-cni to linkerd-cni
* Fixup Dockerfile and clean up code a bit as well as logging statements.
* Update Gopkg.lock after master merge.
* Update test file to remove temporary tag.
* Fixed the command to run during the test while building up the docker run.
* Added attributions to applicable files; in the test file, use a different container for each test scenario and also print the docker logs to stdout when there is an error;
* Add the --no-init-container flag to install and inject. This flag will not output the initContainer and will add an annotation assuming that the cni will be used in this case.
* Update .travis.yml to build the cni-plugin docker image before running the tests.
* Workaround golint warnings.
* Create a new command to install the linkerd-cni plugin.
* Add the --no-init-container option to linkerd inject
* Use the setup ip tables annotation during the proxy auto inject webhook prevent/allow addition of an init container; move cni-plugin tests to the integration-test section of travis
* gate the cni-plugin tests with the -integration-tests flag; remove unnecessary deployment .yaml file.
* Incorporate PR Cleanup suggestions.
* Remove the SetupIPTablesLabel annotation and use config flags and the presence of the init container to determine whether the cni-plugin writes ip tables.
* Fix a logic bug in the cni-plugin code that prevented the iptables from being written; Address PR comments; make tests pass.
* Update go deps shas
* Changed the single file install-cni plugin filename to be .conf vs .conflist; Incorporated latest PR comments around spacing with the new renderer among others.
* Fix an issue with renaming .conf to .conflist when needed.
* Renamed some of the variables to try to make it more clear what is going on.
* Address final PR comments.
* Hide cni flags for the time being.
Signed-off-by: Cody Vandermyn <cody.vandermyn@nordstrom.com>
This picks up the following proxy commits:
* eaabc48 Update tower-grpc
* e9561de Update h2 to 0.1.16
* 28fd5e7 Add Route timeouts (linkerd/linkerd2-proxy#165)
* 5637372 Re-flag tcp_duration tests as flaky
* 20cbd18 Revise several log levels and messages (linkerd/linkerd2-proxy##177)
* ae16978 Remove flakiness from 'profiles' tests
* 49c29cd canonicalize: Only log errors at the WARN level when falling back (linkerd/linkerd2-proxy#174)
* 486dd13 Make outbound router honor `l5d-dst-override` header (linkerd/linkerd2-proxy#173)
* 7adc50d Make timeouts for canonicalization DNS queries tuneable (linkerd/linkerd2-proxy#175)
* 3188179 Try reducing CI flakiness by reducing RUST_TEST_THREADS to 1
Some of these changes will probably need changelog entries:
* Improve logging when rejecting malformed HTTP/2 pseudo-headers
(carllerche/h2#347)
* Improve logging for gRPC errors (tower-rs/tower-grpc#111)
* Add Route timeouts (linkerd/linkerd2-proxy#165)
* Downgrade several of the noisiest log messages to TRACE
(linkerd/linkerd2-proxy##177)
* Add an environment variable for configuring the DNS canonicalization
timeout (linkerd/linkerd2-proxy#175)
* Make outbound router honor `l5d-dst-override` header
(linkerd/linkerd2-proxy#173)
Perhaps all the logging related changes can be grouped into one
changelog entry, though...
Signed-off-by: Eliza Weisman <eliza@buoyant.io>
`fast-build` was performing a full Docker build minus the cli, and then
building cli locally. Separately, shasum was called with a `-p flag,
breaking some builds on Darwin.
Instead, rename `fast-build` to `build-cli-bin`, and restrict it to only
building the cli locally, without any Docker dependencies. Also modify
`bin/linkerd` to call `build-cli-bin` rather than
`docker-build-cli-bin`.
To perform an equivalent of `fast-build`:
`LINKERD_LOCAL_BUILD_CLI=1 bin/docker-build`
`shasum` fix cribbed from #2071.
Relates to #1704
Signed-off-by: Andrew Seigner <siggy@buoyant.io>
- Update h2 to 0.1.15 (linkerd/linkerd2-proxy#172)
carllerche/h2#338 fixes a deadlock in stream reference counts that could
potentially impact the proxy. linkerd/linkerd2-proxy@6df55c0 updates our
`h2` dependency to a version which includes this change.
Signed-off-by: Eliza Weisman <eliza@buoyant.io>
JavaScript assets could be cached across Linkerd releases, showing an
out of date ui, or a broken page.
Modify the webpack build pipeline to add a hash to the JS bundle
filename. Move all logic around webpack-dev-server state from Go into
JS, via a templatized index_bundle.js file, generated at build time.
Disable caching of index_bundle.js in Go, via a `Cache-Control` header.
Fixes#1996
Signed-off-by: Andrew Seigner <siggy@buoyant.io>
The default integration test namespace was `linkerd`.
Modify this default to `l5d-integration`, to minimize risk of
conflicting with an existing linkerd installation.
Signed-off-by: Andrew Seigner <siggy@buoyant.io>
Commit 1: Enable lint check for comments
Part of #217. Follow up from #1982 and #2018.
A subsequent commit will fix the ci failure.
Commit 2: Address all comment-related linter errors.
This change addresses all comment-related linter errors by doing the
following:
- Add comments to exported symbols
- Make some exported symbols private
- Recommend via TODOs that some exported symbols should should move or
be removed
This PR does not:
- Modify, move, or remove any code
- Modify existing comments
Signed-off-by: Andrew Seigner <siggy@buoyant.io>
commit 68f42c337f2580f3b33ddab2e01540f6849d0d1a (HEAD -> master, origin/master)
Author: Oliver Gould <ver@buoyant.io>
Date: Tue Dec 4 07:45:20 2018 -0800
Log discovery updates in the outbound proxy (#153)
When debugging issues that users believe is related to discovery, it's
helpful to get a narrow set of logs out to determine whether the proxy
is observing discovery updates.
With this change, a user can inject the proxy with
```
LINKERD2_PROXY_LOG='warn,linkerd2_proxy=info,linkerd2_proxy::app::outbound::discovery=debug'
```
and the proxy's logs will include messages like:
```
DBUG voting-svc.emojivoto.svc.cluster.local:8080 linkerd2_proxy::app::outbound::discovery adding 10.233.70.98:8080
DBUG voting-svc.emojivoto.svc.cluster.local:8080 linkerd2_proxy::app::outbound::discovery removing 10.233.66.36:8080
```
This change also turns-down some overly chatty INFO logging in main.
A Tap integration test fails and has been fixed by
linkerd/linkerd2-proxy#152.
This change bumps the proxy version to get this change, as well as an
upgrade to the `h2` library for bugfixes.
This change alters the controller's Tap service to include route labels
when translating tap events, modifies the public API to include route
metadata in responses, and modifies the tap CLI command to include
rt_ labels in tap output (when -o wide is used).
The `linkerd` routes command only supports outbound metrics queries (i.e. ones with the `--from` flag). Inbound queries (i.e. ones without the `--from` flag) never return any metrics.
We update the proxy version and use the new canonicalized form for dst labels to gain support for inbound metrics as well.
Signed-off-by: Alex Leong <alex@buoyant.io>
When requests from a pod send requests to itself, the proxy properly redirects traffic from the originating container in the pod through the outbound listener of the proxy. Once the request ends on the inbound side of the proxy, it skips the proxy and calls the original container that made the request. This can cause problems for containers that serve HTTP as the proxy naively tries to initiate an HTTP/2 connection to the destination of a request. (See #1585 for a concrete example)
This PR adds a new iptable rule, coupled with a proxy [change](https://github.com/linkerd/linkerd2-proxy/pull/122) ensure that requests from a that occur in the aforementioned scenario, always redirect to the inbound listener of the proxy first.
fixes#1585
Signed-off-by: Dennis Adjei-Baah <dennis@buoyant.io>
Jest is faster, has more flexibility to run a subset of the tests, and will allow
us to remove a bunch of our assertion libraries.
Many thanks to @grampelberg for prior work on this (#1000)
This PR:
- changes the test runner from karma to jest
- moves individual tests from /test/ to/js/components` where jest expects them
To support reading and writing of the ServiceProfile custom resource, we add a codegen'd Kubernetes client for this resource.
* Adding the ServiceProfile type and related boilerplate to /controller/gen/apis/serviceprofile. This boilerplate also contains directives that control how codegen works.
* A script in /hack which invokes codegen that generates Kubernetes client machinery for interacting with ServiceProfile resources. The majority of the generated code lives in /controller/gen/client.
* The above-mentioned generated code.
Signed-off-by: Alex Leong <alex@buoyant.io>
This PR updates the proxy SHA the build is pinned. This is in order to
track dependency updates in the proxy for the upcoming edge release.
Signed-off-by: Eliza Weisman <eliza@buoyant.io>
Pin the proxy version to a specific SHA instead of floating on latest. This allows breaking changes in the proxy repo to not break the main Linkerd 2 repo.
Signed-off-by: Alex Leong <alex@buoyant.io>
* Upgrade to dep 0.5.0, go 1.10.3
* Remove existing dep binary if it's the wrong version
* Add version in filename of dep binary to prevent version conflicts
Signed-off-by: Kevin Lingerfelt <kl@buoyant.io>
* bin/test-cleanup: Delete test namespaces in paralllel.
My kubectl (1.9.4) waits until the `kubectl delete` operation completes,
unlike previous versions which issue the deletion request and then
immediate return. As a result of this change, bin/test-cleanup become
much slower since each deletion was happening serially. Fix this by
issuing all the deletions in parallel with a single `kubectl delete`.
Signed-off-by: Brian Smith <brian@briansmith.org>
Without this change the Linux build fails. Apparently
commit 554ffe6a46 added the use of a
bash-specific feature.
Signed-off-by: Brian Smith <brian@briansmith.org>
* Allow docker-build-proxy to override the proxy version
* Update based on review feedback
* fetch-proxy should return full path to executable
Signed-off-by: Kevin Lingerfelt <kl@buoyant.io>
* Stop using `installsuffix` when building Go code.
See https://plus.google.com/117192131596509381660/posts/eNnNePihYnK.
`-installsuffix cgo` isn't necessary as of Go 1.10 (where build caching
changed substantially) and it probably wasn't necessary earlier.
Signed-off-by: Brian Smith <brian@briansmith.org>
* Add flag that skips `dep ensure` to bin/fast-build
bin/fast-build is supposed to be fast. `dep ensure -vendor-only` is too slow
to meet this goal. Add `LINKERD_SKIP_DEP` to allow skipping it. The default
behavior is kept as-is to reduce new users' confusion.
The difference in speed isn't too notable now because the bin/docker-build
step drowns out the win currently. But if/when the bin/docker-build step is
replaced, this matters a lot.
Signed-off-by: Brian Smith <brian@briansmith.org>
The control-plane's `ClusterRole` and `ClusterRoleBinding` objects are
global. Because their names did not vary across multiple control-plane
deployments, it prevented multiple control-planes from coexisting (when
RBAC is enabled).
Modify the `ClusterRole` and `ClusterRoleBinding` objects to include the
control-plane's namespace in their names. Also modify the integration
test to first install two control-planes, and then perform its full
suite of tests, to prevent regression.
Fixes#1292.
Signed-off-by: Andrew Seigner <siggy@buoyant.io>
This PR begins to migrate Conduit to Linkerd2:
* The proxy has been completely removed from this repo, and is now located at
github.com/linkerd/linkerd2-proxy.
* A `Dockerfile-proxy` has been added to fetch the most-recently published proxy
binary from build.l5d.io.
* Proxy-specific protobuf bindings have been moved to
github.com/linkerd/linkerd2-proxy-api.
* All docker images now use the gcr.io/linkerd-io registry.
* `inject` now uses `LINKERD2_PROXY_` environment variables
* Go paths have been updated to reflect the new (future) repo location.
The proxy's metrics are instrumented with a `tls` label that describes
the state of TLS for each connection and associated messges.
This same level of detail is useful to get in `tap` output as well.
This change updates Tap in the following ways:
* `TapEvent` protobuf updated:
* Added `source_meta` field including source labels
* `proxy_direction` enum indicates which proxy server was used.
* The proxy adds a `tls` label to both source and destination meta indicating the state of each peer's connection
* The CLI uses the `proxy_direction` field to determine which `tls` label should be rendered.
PR #978 introduced usage of parallel in docker-build. Unfortunately this
breaks if the system has non-GNU parallel.
Remove usage of parallel until we can do at least one of the following:
- detect version of parallel installed
- make usage of parallel optional and off by default
- confirm this speeds up builds for a majority of use cases
Signed-off-by: Andrew Seigner <siggy@buoyant.io>
* Enable optional parallel build of docker images
By default, docker does image builds in a single thread. For our containers, this is a little slow on my system. Using `parallel` allows for *optional* improvements in speed there.
Before: 41s
After: 22s
* Move parallel help text to stderr
* Start running integration tests in CI
* Add gcp helper funcs
* Split integration test cleanup into separate phase
Signed-off-by: Kevin Lingerfelt <kl@buoyant.io>
This PR changes the proxy's Inotify watch code to avoid always falling back to
polling the filesystem when the watched files don't exist yet. It also contains
some additional cleanup and refactoring of the inotify code, including moving
the non-TLS-specific filesystem watching code out of the `tls::config` module
and into a new `fs_watch` module.
In addition, it adds tests for both the polling-based and inotify-based watch
implementations, and changes the polling-based watches to hash the files rather
than using timestamps from the file's metadata to detect changes. These changes
are originally from #1094 and #1091, respectively, but they're included here
because @briansmith asked that all the changes be made in one PR.
Closes#1094. Closes#1091. Fixes#1090. Fixes#1097. Fixes#1061.
Signed-off-by: Eliza Weisman <eliza@buoyant.io>
protobuf has a `go_package` option that can be used to explicitly name
Go packages such that they can be imported without additional rewrites.
This allows us to store proto files without additional, redundant
directories (which were used for packaging hints, previously).
This change adds an explicit `go_package` to all .proto files and
updates `bin/protoc-go.sh` to ensure these packages are output into
$GOPATH (so that the go_package can be absolute). This removes the need
to manually rewrite imports in bin/protoc-go.sh.
When developing features in the proxy, that rely on Linux-only OS features,
developers using other operating systems may find it inconvenient to test
their changes. While we run CI builds on Linux, and may have access to Linux
testing environments, this is not as tightly integrated into the proxy
development workflow as running a quick `cargo test` on the host OS.
For example, I found it inconvenient to test the `inotify` based filesystem
watch code I've been adding in recent commits, and had to do things like
opening a WIP PR for a branch to get CI to run the tests. This workflow is not
ideal.
This PR adds an (admittedly somewhat hacky) script and Dockerfile for running
the proxy's tests in Docker. This accomplishes approximately the same goal as
the `PROXY_SKIP_TESTS` flag that we used to have, but with the advantage that
we no longer include the test dependencies in release builds.
Of course, this also means that we no longer share any of the dependencies
between the test docker build and the release docker build, which is a shame.
It might be worthwhile to re-introduce a dependencies image so that cached
builds of the proxy's dependencies can be shared between the test and release
Dockerfiles. However, I thought that deserved to be discussed separately from
the changes I made in this branch.
Signed-off-by: Eliza Weisman <eliza@buoyant.io>
* Handle an edge case when using bin/web
There's a weird error running `bin/web dev` if you don't have conduit installed on a kubernetes cluster. Nothing in the docs mention that you need to work on this.
Output a user friendly error when we can't find a pod and update the docs to remind folks to install conduit first. Fixes#1070
* Wrap text, send to stderr, fail when missing
* Update web dockerfile to use dev deps when building prod assets
* Don't re-run yarn install as pre-req for build/run/test
Signed-off-by: Kevin Lingerfelt <kl@buoyant.io>
As part of trying to be fancy, I moved the `setup` step into build. This breaks the docker builds because we need to run yarn *without* NODE_ENV=production and then the build *with* NODE_ENV=production (to do things like minify/compress assets).
Split apart build as something without setup and provide a default target that does setup + build for travis.
- Switched from `es2015` to `env` for the default preset. This is the recommended preset and allows us to track the latest and greatest moving forward.
- Added `react-app` as a preset. We get class properties (and thus => for context) as well as the current recommended settings for react apps.
- Created a `web` script that provides functions for common tasks. `react-app` requires that BABEL_ENV/NODE_ENV is set and this guarantees it.
- Updated the web dockerfile to set NODE_ENV correctly and use `bin/web`.
- Moved the babel related modules over to devDependencies.
The proxy's Dockerfile is split into stages: build and runtime.
The build stage includes all of the intermdiate build information, and
the runtime image discards these layers with a small production-ready
image.
In order to improve docker build times, we can save this build layer to
be reused.
This reduces the docker build of the proxy in CI from 15 minutes to
about 7.5 minutes (when the proxy is not changed).
The proxy Dockerfile includes test execution. While the intentions of
this are good, it has unintended consequences: we can ship code linked
with test dependencies.
Because we have other means for testing proxy code (cargo, locally; and
CI runs tests outside of Docker), it is fine to remove these tests.
* Remove the telemetry service
The telemetry service is no longer needed, now that prometheus scrapes
metrics directly from proxies, and the public-api talks directly to
prometheus. In this branch I'm removing the service itself as well as
all of the telemetry protobuf, and updating the conduit install command
to no longer install the service. I'm also removing the old version of
the stat command, which required the telemetry service, and renaming the
statsummary command to stat.
* Fix time window tests
* Remove deprecated controller scrape config
Signed-off-by: Kevin Lingerfelt <kl@buoyant.io>
Using a vanilla Grafana Docker image as part of `conduit install`
avoided maintaining a conduit-specific Grafana Docker image, but made
packaging dashboard json files cumbersome.
Roll our own Grafana Docker image, that includes conduit-specific
dashboard json files. This significantly decreases the `conduit install`
output size, and enables dashboard integration in the docker-compose
environment.
Fixes#567
Part of #420
Signed-off-by: Andrew Seigner <siggy@buoyant.io>
* Add tests/utils/scripts for running integration tests
Add a suite of integration tests in the `test/` directory, as well as
utilities for testing in the `testutil/` directory.
You can use the `bin/test-run` script to run the full suite of tests,
and the `bin/test-cleanup` script to cleanup after the tests.
The test/README.md file has more information about running tests.
@pcalcado, @franziskagoltz, and @rmars also contributed to this change.
* Create TEST.md file at the root of the repo
* Update based on review feedback
* Relax external service IP timeout for GKE
* Update TEST.md with more info about different types of test runs
* More updates to TEST.md based on review feedback
Signed-off-by: Kevin Lingerfelt <kl@buoyant.io>
CI builds on master have been failing to publish `cli-bin` images because the
`docker-push` script still refers to the `cli` image, though it was removed in
e7c4a9d4b9.
This change removes references to the `cli` image from all scripts.
This image isn't used. It references its base image using the `latest` tag, which
is wrong; it should have been using the tag that the base image was built with. It
is likely that the last few iterations of this image that we've published have
wrong and useless contents.
With that in mind, just remove the image.
Fixes#578.
Signed-off-by: Brian Smith <brian@briansmith.org>
The build scripts assume they are executed from the root of this repo.
This prevents running scripts from other locations, for example,
`cd web && ../bin/go-run .`.
Modify the build scripts to work regardless of current directory.
Fixes#301
Signed-off-by: Andrew Seigner <siggy@buoyant.io>
* Update go-run to set version equal to root-tag
* Fix inject tests for undefined version change
* Pass inject version explitictly as arg
Signed-off-by: Kevin Lingerfelt <kl@buoyant.io>
The current proxy Dockerfile configuration does not cache dependencies
well, which can increase build times substantially.
By carefully splitting proxy/Dockerfile into several stages that mock
parts of the project, dependencies may be built and cached in Docker
such that changes to the proxy only require building the conduit-proxy
crate.
Furthermore, proxy/Dockerfile now runs the proxy's tests before
producing an artifact, unless the ` PROXY_SKIP_TESTS` build-arg is set
and not-empty.
The `PROXY_UNOPTIMIZED` build-arg has been added to support quicker,
debug-friendly builds.
Previously we didn't verify that the downloaded dep binary is the right
binary.
Verify that the downloaded binary is correct.
Signed-off-by: Brian Smith <brian@briansmith.org>
The logic for choosing the 32-bit vs. 64-bit version of dep was
inverted.
Fix this by simply always using the 64-bit version.
Signed-off-by: Brian Smith <brian@briansmith.org>
* Add `bin/dep` which fetches a fixed version of `dep` to be used.
* Upgrade from dep 0.3.1 to 0.4.1
* Fix inconsistent Gopkg.lock by checking in the result of `bin/dep ensure`
Signed-off-by: Alex Leong <alex@buoyant.io>
The conduit.io/* k8s labels and annotations we're redundant in some
cases, and not flexible enough in others.
This change modifies the labels in the following ways:
`conduit.io/plane: control` => `conduit.io/controller-component: web`
`conduit.io/controller: conduit` => `conduit.io/controller-ns: conduit`
`conduit.io/plane: data` => (remove, redundant with `conduit.io/controller-ns`)
It also centralizes all k8s labels and annotations into
pkg/k8s/labels.go, and adds tests for the install command.
Part of #201
Signed-off-by: Andrew Seigner <siggy@buoyant.io>
* Set conduit version to match conduit docker tags
Signed-off-by: Kevin Lingerfelt <kl@buoyant.io>
* Remove --skip-inbound-ports for emojivoto
Signed-off-by: Kevin Lingerfelt <kl@buoyant.io>
* Rename git_sha => git_sha_head
Signed-off-by: Kevin Lingerfelt <kl@buoyant.io>
* Switch to using the go linker for setting the version
Signed-off-by: Kevin Lingerfelt <kl@buoyant.io>
* Log conduit version when go servers start
Signed-off-by: Kevin Lingerfelt <kl@buoyant.io>
* Cleanup conduit script
Signed-off-by: Kevin Lingerfelt <kl@buoyant.io>
* Add --short flag to head sha command
Signed-off-by: Kevin Lingerfelt <kl@buoyant.io>
* Set CONDUIT_VERSION in docker-compose env
Signed-off-by: Kevin Lingerfelt <kl@buoyant.io>
If docker image tags were out of date, ci would not fail until the
docker-deploy stage (master merge).
Modify ci to validate tags as part of the default ci run.
Signed-off-by: Andrew Seigner <siggy@buoyant.io>
DOCKER_FORCE_BUILD, combined with symbolic tags, added complexity and
risk of running unintended versions of the code.
This change removes DOCKER_FORCE_BUILD, and sets all Docker tags
programmatically. The decision to pull or build has been moved up the
stack from _docker.sh to the docker-build-* scripts. Workflows that
want to favor docker pulls (like ci), can do so explicitly via
docker-pull.
fixes#141
Signed-off-by: Andrew Seigner <andrew@sig.gy>
Previously, proxy-deps and go-deps included the source tree for local
projects. This can cause build conflicts when files are renamed.
By adopting a multi-stage build for the proxy-deps image, we can be sure
that we only preserve essential dependencies & manifests in the
proxy-deps and go-deps images.
Furthermore, `bin/update-go-deps-shas` and `bin/update-proxy-deps-shas` have
been added to ease maintenance when files are changed.
Fixes#159
Signed-off-by: Oliver Gould <ver@buoyant.io>
* Move healthcheck proto to separate file, use throughout
Signed-off-by: Kevin Lingerfelt <kl@buoyant.io>
* Remove Check message from healthcheck.proto
Signed-off-by: Kevin Lingerfelt <kl@buoyant.io>
* Standardize healthcheck protobuf import name
Signed-off-by: Kevin Lingerfelt <kl@buoyant.io>
Previously if dependencies changed but dep image SHAs were not updated,
the build could succeed, creating docker images with indeterminate
dependencies.
This change checks the dependency image SHAs hard-coded in Dockerfile's
against the current source tree. If the SHAs do not match, the build
fails.
Fixes#118
Signed-off-by: Andrew Seigner <andrew@sig.gy>
The image tags for gcr.io/runconduit/go-deps and
gcr.io/runconduit/proxy-deps were not updating to account for all
changes in those images.
Modify SHA generation to include all files that affect the base
dependency images. Also add instructions to README.md for updating
hard-coded SHAs in Dockerfile's.
Fixes#115
Signed-off-by: Andrew Seigner <andrew@sig.gy>
We’ve built Conduit from the ground up to be the fastest, lightest,
simplest, and most secure service mesh in the world. It features an
incredibly fast and safe data plane written in Rust, a simple yet
powerful control plane written in Go, and a design that’s focused on
performance, security, and usability. Most importantly, Conduit
incorporates the many lessons we’ve learned from over 18 months of
production service mesh experience with Linkerd.
This repository contains a few tightly-related components:
- `proxy` -- an HTTP/2 proxy written in Rust;
- `controller` -- a control plane written in Go with gRPC;
- `web` -- a UI written in React, served by Go.
Andrew Seigner
Alejandro Pedraza
陈谭军
Ivan Sim
Alex Leong
Oliver Gould
Eliza Weisman
Kevin Leimkuhler
Dennis Adjei-Baah
Carol A. Scott
Kevin Leimkuhler
Andrew Seigner
Thomas Rampelberg
Kevin Lingerfelt
Alejandro Pedraza
Cody Vandermyn
Risha Mars
Brian Smith
Alena Varkockova