Commit Graph

2636 Commits

Author SHA1 Message Date
Alex Leong 5d10176d9b
Rename --addon-config flag to --config (#5075)
The `--addon-config` flag allows users to supply a yaml config file which will override the Values used to install or upgrade Linkerd.  While this is useful for supplying config for addons, it can be used to configure any part of the Values struct.  Thus, we rename the flag to `--config`.

Signed-off-by: Alex Leong <alex@buoyant.io>
2020-10-13 14:07:18 -07:00
Tarun Pothulapati 2a5e7dba62
Handle grafana add-on config repair (#5059)
* Handle grafana add-on config repair

Fixes #5014

In Grafana Add-On, Default fields i.e `grafana.image.name`, `grafana.name`
have been removed from `linkerd-config-addons` after `2.8.1`. Only
overriden values are stored in `linkerd-config-addons` as of now.
Hence, `grafana.image.name` has to be removed from
`linkerd-config-addons` unless they are overriden so that updates
to it can take place especially the move from `gcr` to `ghcr`.

This also removes `grafana.name` field if they are set to default, as
its removed.

This problem will not occur again even if we update default values, as
default values are not stored in `linekrd-config-addons` anymore for all
add-ons.

Signed-off-by: Tarun Pothulapati <tarunpothulapati@outlook.com>
2020-10-13 13:12:49 -07:00
Alex Leong 0f4be79830
Remove install, global, and proxy fields from config entirely (#5067)
Remove the now unused `global`, `install`, and `proxy` fields from the `linkerd-config` configmap.

Signed-off-by: Alex Leong <alex@buoyant.io>
2020-10-13 09:12:32 -07:00
Markus f8e7295bdc
MC chart load balancer ip (#5048)
adding loadBalancerIP to linkerd2-multicluster chart

Sometimes you are in need to tell the gateway service to pick up / request a specific IP from the LB.
e.g. when you talk to another cluster that is having another firewall in front and not permitting access from random IPs.

Solution
Minor change in the chart for Multicluster.

Validation
Example in a GKE:
Register a static IP, note it. Then

helm install linkerd-mc linkerd2/linkerd2-multicluster --set loadBalancerIP="<IP>"

Your gateway service will come up with the IP you have given it.
If you don't set the parameter, then the LB will give out a random IP.

If you don't have a cluster, look at the yaml produced by helm template...
and look if the loadBalancerIP: <IP>  is there

```
`apiVersion: v1
kind: Service
.
.
.
  selector:
    app: linkerd-gateway
  type: LoadBalancer
  loadBalancerIP: 1.1.1.1`
```

Signed-off-by: Markus Bettsteller <markus@bettsteller.de>
2020-10-13 18:08:40 +03:00
Alex Leong 41c1fc65b0
Upgrade using config overrides (#5005)
This is a major refactor of the install/upgrade code which removes the config protobuf and replaces it with a config overrides secret which stores overrides to the values struct.  Further background on this change can be found here: https://github.com/linkerd/linkerd2/discussions/4966

Note: as-is this PR breaks injection.  There is work to move injection onto a Values-based config which must land before this can be merged.

A summary of the high level changes:

* the install, global, and proxy fields of linkerd-config ConfigMap are no longer populated
* the CLI install flow now follows these simple steps:
  * load default Values from the chart
  * update the Values based on the provided CLI flags
  * render the chart with these values
  * also render a Secret/linkerd-config-overrides which describes the values which have been changed from their defaults
* the CLI upgrade flow now follows these simple stesp:
  * load the default Values from the chart
  * if Secret/linkerd-config-overrides exists, apply the overrides onto the values
  * otherwise load the legacy ConfigMap/linkerd-config and use it to updates the values
  * further update the values based on the provided CLI flags
  * render the chart and the Secret/linkerd-config-overrides as above
* Helm install and upgrade is unchanged

Signed-off-by: Alex Leong <alex@buoyant.io>
2020-10-12 14:23:14 -07:00
Oliver Gould 4d7da72192
proxy: v2.114.0 (#5062)
This release overhauls the discovery and routing logic implemented by
the proxy: instead of looking at HTTP request metadata for service
discovery, the outbound proxy now exclusively use each connection's
target IP:PORT. This eager resolution eliminates per-request cache
binding; and supports using TrafficSplit with non-HTTP services.

This has a few side effects:

- The `l5d-dst-override` header is no longer honored.
- When the application attempts to connect to a pod IP, the proxy no
  longer load balances these requests among all pods in the service.
  The proxy will now honor session-stickiness as selected by an
  application-level load balancer.
- `TrafficSplits` are only applied when a client targets a service's IP.
- The proxy no longer performs DNS "canonicalization" to translate
  relative host header names to a fully-qualified form.

---

* Unify RequestFilter and Admit middlewares (linkerd/linkerd2-proxy#692)
* Only allow name-based profile discovery for inbound requests (linkerd/linkerd2-proxy#695)
* outbound: initial tests for TCP mTLS (with fewer moving parts) (linkerd/linkerd2-proxy#693)
* Stop honoring DESTINATION_GET_* configuration (linkerd/linkerd2-proxy#696)
* stack: add SwitchReady service (linkerd/linkerd2-proxy#694)
* telemetry: Remove trailing comma in build_info labels (linkerd/linkerd2-proxy#699)
* Update Rust to 1.47.0 (linkerd/linkerd2-proxy#701)
* cache: Delete benchmarks (linkerd/linkerd2-proxy#705)
* outbound: Discover profiles for each unique TCP target (linkerd/linkerd2-proxy#704)
2020-10-12 12:01:51 -07:00
Zahari Dichev 60d8f34095
avoid waiting when creating calico cluster with kind (#5064)
Currently the --wait flag times out when creating a calico cluster. The result is that we end up waiting for 5 minutes to simply emit a warning and continue. Instead we can check the readiness of some k8s components to ensure our cluster is up and running and avoid the delay.

Signed-off-by: Zahari Dichev zaharidichev@gmail.com
2020-10-12 18:26:00 +03:00
Alejandro Pedraza 777b06ac55
Expand 'linkerd edges' to work with TCP connections (#5040)
* Expand 'linkerd edges' to work with TCP connections

Fixes #4999

Before:
```
$ bin/linkerd edges po -owide
SRC                                   DST                                    SRC_NS    DST_NS    CLIENT_ID   SERVER_ID   SECURED
linkerd-prometheus-764ddd4f88-t6c2j   rabbitmq-controller-5c6cf7cc6d-8lxp2   linkerd   default                           √
linkerd-prometheus-764ddd4f88-t6c2j   temp                                   linkerd   default                           √

```

After:
```
$ bin/linkerd edges po -owide
SRC                                   DST                                    SRC_NS    DST_NS    CLIENT_ID         SERVER_ID         SECURED
temp                                  rabbitmq-controller-5c6cf7cc6d-5fpsc   default   default   default.default   default.default   √
linkerd-prometheus-66fb97b7fc-vpnxf   rabbitmq-controller-5c6cf7cc6d-5fpsc   linkerd   default                                       √
linkerd-prometheus-66fb97b7fc-vpnxf   temp                                   linkerd   default                                       √
```

With the latest proxy upgrade to v2.113.0 (#5037), the `tcp_open_total` metric now contains the `client_id` label so that we can replace the http-only metric `response_total` with this one to determine edges for TCP-only connections.

This change basically performs the same query as before, but two times, one for `response_total` and another for `tcp_open_total`. For each resulting entry, the latter is kept if `client_id` is present, otherwise the former is used (if present at all). That way things keep on working for older proxies.

Disclaimers:
- This doesn't fix #3706: if two sources connect to the same destination there's no way to tell them appart from the metrics perspective and their edges can get mangled. To fix that, the proxy would have to expose `src_resource` labels in the `tcp_open_total` total inbound metric.
- Note connections coming from prometheus are still unidentified. The reason is those hit the proxy's admin server (instead of the main container) which doesn't expose metrics.
2020-10-12 09:14:39 -05:00
Alejandro Pedraza 3af25fa886
Fix how env vars are set in CI (#5054)
Replaced `set-env` directives with environment files, as explained
[here](https://github.blog/changelog/2020-10-01-github-actions-deprecating-set-env-and-add-path-commands/)

This gets rids of warnings of the sort:
```
The `set-env` command is deprecated and will be disabled soon. Please
upgrade to using Environment Files. For more information see:
https://github.blog/changelog/2020-10-01-github-actions-deprecating-set-env-and-add-path-commands/
```
2020-10-09 19:24:41 -07:00
Chris Campbell 8be08994f0
Add HP to ADOPTERS.md (#5061)
Signed-off-by: Chris Campbell <chrs.cmpbl@gmail.com>
2020-10-09 18:42:18 -07:00
Alex Leong 530d8beccc
Add podLabels and podAnnotations to Values struct (#5056)
PR https://github.com/linkerd/linkerd2/pull/5027 added `podLabels` and `podAnnotations` to `values.yaml` to allow setting labels and annotations on pods in the Helm template.  However, these fields were not added to the `Values` struct in `Values.go`.  This means that these fields were not serialized out to the `linkerd-config` or to the `linkerd-config-overrides`.  Furthermore, in PR #5005 which moves to using the `Values` struct more authoritatively, the `podLabels` and `podAnnotations` fields would not take effect at all.

Add these fields to the `Values` struct and update all test fixtures accordingly.

Signed-off-by: Alex Leong <alex@buoyant.io>
2020-10-09 09:27:28 -07:00
Oliver Gould 5f694513bd
bin/tests: Improve argument parsing (#5060)
The `bin/tests` script takes command-line arguments, but it requires
that all arguments are specified before the linkerd binary path; and it
silently ignores flags that follow the linkerd binary. Furthermore,
unexpected flags may be incorrectly parsed as the linkerd binary path.

This changes argument parsing to be more flexible about ordering; and it
prints the full usage error when unexpected flags are encountered.
2020-10-09 07:27:22 -07:00
Raphael Taylor-Davies 2a97d76121
Expose podLabels and podAnnotations in helm chart (#5027)
This adds the `podAnnotations` and `podLabels` values in `values.yml` for adding custom annotations/labels to all the control plane pods.

Closes (#5025)

Signed-off-by: Raphael Taylor-Davies <r.taylordavies@googlemail.com>
2020-10-08 16:46:53 -05:00
Alejandro Pedraza 11a5d1d427
Fix Heartbeat mem and cpu stats (#5042)
Since k8s 1.16 cadvisor uses the `container` label instead of
`container_name` in the prometheus metrics it exposes.
The heartbeat queries were using the latter, so they were broken
for k8s version since 1.16.

Note that the `p99-handle-us` value is still missing because the
`request_handle_us` metrics is always zero.
2020-10-08 16:31:16 -05:00
Zahari Dichev ffa7157907
Add warnings for failed secred mounts to expect warnings in it (#5053)
Seems that Helm is cleaning orphaned resources. Pods that depend on them seem to be not upgraded on time, causing some warnings to be emitted and he CI process to fail

Signed-off-by: Zahari Dichev <zaharidichev@gmail.com>
2020-10-08 15:06:03 -05:00
Zahari Dichev c1bc91dc9c
edge-20.10.2 (#5050)
## edge-20.10.2

This edge release adds more improvements for mTLS for all TCP traffic.
It also includes significant internal improvements to the way Linkerd
configuration is stored within the cluster.

* Changed TCP metrics exported by the proxy to ensure that peer
  identities are encoded via the `client_id` and `server_id` labels.
* Removed the dependency of control plane components on `linkerd-config`
* Updated the data structure `proxy-injector` uses to derive the configuration
  used when injecting workloads
2020-10-08 20:18:07 +03:00
Tarun Pothulapati 1e7bb1217d
Update Injection to use new linkerd-config.values (#5036)
This PR Updates the Injection Logic (both CLI and proxy-injector)
to use `Values` struct instead of protobuf Config, part of our move
in removing the protobuf.

This does not touch any of the flags, install related code.

Signed-off-by: Tarun Pothulapati <tarunpothulapati@outlook.com>

Co-authored-by: Alex Leong <alex@buoyant.io>
2020-10-07 09:54:34 -07:00
Tarun Pothulapati 5e774aaf05
Remove dependency of linkerd-config for control plane components (#4915)
* Remove dependency of linkerd-config for most control plane components

This PR removes the dependency of `linkerd-config` into control
plane components by making all that information passed through CLI
flags. As most of these components require a couple of flags, passing
them as flags could be more helpful, as updations to the flags trigger a
rollout unlike a configMap update.

This does not update the proxy-injector as it needs a lot more data
and mounting `linkerd-config` is better.
2020-10-06 22:19:18 +05:30
Alex Leong 34a35c612d
Add flag library (#5038)
One of the challenges with using the Cobra flag library is that it is difficult to differentiate between a flag that has been set explicitly to its default value and one that has been unset (and thus remains at its default value).  In particular, when processing CLI flags for the purposes of install and upgrade, this distinction is important because we want to persist the effects of flags which have been set.

Flag is an interface which describes a command line flag that affects the Helm Values used to render Helm charts.  This interface allows us to iterate over flags which have been set and apply their effects to the Values.  To see how this library is to be used, see https://github.com/linkerd/linkerd2/pull/5005

Signed-off-by: Alex Leong <alex@buoyant.io>
2020-10-05 14:57:40 -07:00
Oliver Gould 127791c362
proxy: v2.113.0 (#5037)
This release includes changes to TCP metrics to ensure that peer
identities are encoded via the `client_id` and `server_id` labels.

---

* outbound: Explicitly ignore the source address for tap (linkerd/linkerd2-proxy#680)
* Update proxy-api and tonic (linkerd/linkerd2-proxy#682)
* http: Lazily build http/tcp stacks (linkerd/linkerd2-proxy#681)
* outbound: Remove required identity from HttpLogical (linkerd/linkerd2-proxy#683)
* profiles: Expose the fully_qualified_name (linkerd/linkerd2-proxy#684)
* request-filter: Support altering the request type (linkerd/linkerd2-proxy#685)
* tracing: Set contexts in new_service/make_service (linkerd/linkerd2-proxy#686)
* discover: Allow resolution streams to terminate (linkerd/linkerd2-proxy#689)
* metrics: add peer identities to all TLS metric labels (linkerd/linkerd2-proxy#687)
* outbound: Return a default endpoint on reject (linkerd/linkerd2-proxy#690)
* Skip endpoint resolution when profile lookup is rejected (linkerd/linkerd2-proxy#691)
2020-10-02 14:42:26 -07:00
Alejandro Pedraza e1772ae183
Fixed releases.yaml by pulling images directly from ghcr.io (#5035)
Previously, `releases.yaml` was trying to load images into the kind
clusters but that failed because those images were already in `ghcr.io`
and not in the local docker cache, but that failure was masked.
Unmasking that failure revealed some flaws that this change addresses:

- In `bin/_test_helpers` (used by `bin/tests`), modified the `images`
arg to accept `docker(default)|archive|skip`, for determining how to
load the images into the cluster (if loading them at all)
- In `bin/image-load`, changed arg `images` to `archive` which is more
descriptive.
- Have `kind_integration.yml` call `bin/tests --images archive`.
- Have `release.yml` call `bin/tests --images skip`.
2020-10-02 08:05:17 -05:00
Alejandro Pedraza 2b7bc7362a
Edge-20.10.1 changes (#5033)
* Edge-20.10.1 changes

## edge-20.10.1

This edge release includes a couple of external contributions towards
improved cert-manager support and Grafana charts fixes, among other
enhancements.

* Changed the type of the injector and tap API secrets to `kubernetes.io/tls`,
  so they can be provisioned by cert-manager (thanks @cypherfox!)
* Fixed the "Kubernetes cluster monitoring" Grafana dashboard that had a few
  charts with incomplete data (thanks @aimbot31!)
* Fixed the `service-mirror` multicluster component so that it retries
  connections to the target cluster's Kubernetes API when it's not reachable,
  instead of blocking
* Increased the proxy's default timeout for DNS resolution to 500ms, as there
  were reports that 100ms was too restrictive

Co-authored-by: Kevin Leimkuhler <kevin@kleimkuhler.com>
2020-10-01 15:35:34 -05:00
Tarun Pothulapati faf77798f0
Update check to use new linkerd-config.values (#5023)
This branch updates the check functionality to read
the new `linkerd-config.values` which contains the full
Values struct showing the current state of the Linkerd
installation. (being added in #5020 )

This is done by adding a new `FetchCurrentConfiguraiton`
which first tries to get the latest, if not falls back
to the older `linkerd-config` protobuf format.`

Signed-off-by: Tarun Pothulapati <tarunpothulapati@outlook.com>
2020-10-01 11:19:25 -07:00
Kevin Leimkuhler 6b7a39c9fa
Set FQN in profile resolutions (#5019)
## Motivation

Closes #5016

Depends on linkerd/linkerd2-proxy-api#44

## Solution

A `profileTranslator` exists for each service and now has a new
`fullyQualifiedName` field.

This field is used to set the `FullyQualifiedName` field of
`DestinationProfile`s each time an update is sent.

In the case that no service profile exists for a service, a default
`DestinationProfile` is created and we can use the field to set the correct
name.

In the case that a service profile does exist for a service, we still use this
field to set the name to keep it consistent.

### Example

Install linkerd on a cluster and run the destination server:

```
go run controller/cmd/main.go destination -kubeconfig ~/.kube/config
```

Get the IP of a service. Here, we'll get the ip for `linkerd-identity`:

```
> kubectl get -n linkerd svc/linkerd-identity
NAME               TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
linkerd-identity   ClusterIP   10.43.161.68   <none>        8080/TCP   4h25m
```

Get the profile of `linkerd-identity` from service name or IP and note the
`FullyQualifiedName` field:

```
> go run controller/script/destination-client/main.go -method getProfile -path 10.43.161.68:8080
INFO[0000] fully_qualified_name:"linkerd-identity.linkerd.svc.cluster.local" ..
```

```
> go run controller/script/destination-client/main.go -method getProfile -path linkerd-identity.linkerd.svc.cluster.local
INFO[0000] fully_qualified_name:"linkerd-identity.linkerd.svc.cluster.local" ..
```

Signed-off-by: Kevin Leimkuhler <kevin@kleimkuhler.com>
2020-10-01 11:06:00 -04:00
Alex Leong 6452fbbdfa
Add values to linkerd-config (#5020)
Fixes #5008 

We add a `values` file to the `ConfigMap/linkerd-config` resource.  This file holds the full Values which were used to render the chart except that private data such as the identity issuer key are redacted.  This file is currently unused but will eventually be used by CLI commands such as `check` and `inject` which need to load Linkerd's configuration (as described in #5009).

This is one step in a larger effort to eventually get rid of the other files in `ConfigMap/linkerd-config`.

Signed-off-by: Alex Leong <alex@buoyant.io>
2020-09-30 11:37:25 -07:00
Oliver Gould d4990a642a
proxy: v2.112.0 (#5003)
This release increases the default timeout for DNS resolution to 500ms,
as there were reports that 100ms was too restrictive.

This also includes several internal changes to facilitate
connection-oriented caching; but these changes are not expected to
impact runtime behavior.

---

* cache: Implement NewService (linkerd/linkerd2-proxy#668)
* Implement fallback for NewService (linkerd/linkerd2-proxy#669)
* discover: Make endpoints with `NewService` (linkerd/linkerd2-proxy#671)
* Implement profile discovery as a NewService (linkerd/linkerd2-proxy#673)
* Make HTTP client stacks synchronous (linkerd/linkerd2-proxy#674)
* Increase default Dns canonicalize timeout (linkerd/linkerd2-proxy#672)
* Update the accept stack to use NewService (linkerd/linkerd2-proxy#675)
* Split prefix buffering from HTTP detection (linkerd/linkerd2-proxy#676)
* transport: Fix server address metadtata (linkerd/linkerd2-proxy#678)
* outbound: Restore spawn-ready (linkerd/linkerd2-proxy#679)
2020-09-29 13:33:28 -07:00
Alex Leong 788479b7b0
Fix upgrade test (#5021)
A conflict between #4911 and #4737 caused unit test to be broken.

#4737 added a new test to `upgrade_test.go` and the changes in
#4911 updated all of these test to ignore differences in the config
overrides secret.  Since these two PRs merged in parallel, the new
test was missing this update.

Update the new test to also ignore differences in the config overrides
secret as the other ones do.

Signed-off-by: Alex Leong <alex@buoyant.io>
2020-09-29 12:41:42 -07:00
aimbot31 7c08fffd8a
Fix kubernetes grafana dashboard (#4380) (#5012)
Prometheus use a relabel rule that changed since 1.16

Use "pod_name" and "pod" to avoid breaking changes.
Also use "container" and "container_name" for the
same reasons.

Fixes #4380

Signed-off-by: Florian Davasse <florian.davasse@stack-labs.com>
2020-09-29 11:28:53 -05:00
Alex Leong 1784f0643e
Add linkerd-config-overrides secret (#4911)
This PR adds a new secret to the output of `linkerd install` called `linkerd-config-overrides`.  This is the first step towards simplifying the configuration of the linkerd install and upgrade flow through the CLI.  This secret contains the subset of the values.yaml which have been overridden.  In other words, the subset of values which differ from their default values.  The idea is that this will give us a simpler way to produce the `linkerd upgrade` output while still persisting options set during install.  This will eventually replace the `linkerd-config` configmap entirely.

This PR only adds and populates the new secret.  The secret is not yet read or used anywhere.  Subsequent PRs will update individual control plane components to accept their configuration through flags and will update the `linkerd upgrade` flow to use this secret instead of the `linkerd-config` configmap.

This secret is only generated by the CLI and is not present or required when installing or upgrading with Helm.

Here are sample contents of the secret, base64 decoded.  Note that identity tls context is saved as an override so that it can be persisted across updates.  Since these fields contain private key material, this object must be a secret.  This secret is only used for upgrades and thus only the CLI needs to be able to read it.  We will not create any RBAC bindings to grant service accounts access to this secret.

```
global:
  identityTrustAnchorsPEM: |
    -----BEGIN CERTIFICATE-----
    MIIBhDCCASmgAwIBAgIBATAKBggqhkjOPQQDAjApMScwJQYDVQQDEx5pZGVudGl0
    eS5saW5rZXJkLmNsdXN0ZXIubG9jYWwwHhcNMjAwODI1MjMzMTU3WhcNMjEwODI1
    MjMzMjE3WjApMScwJQYDVQQDEx5pZGVudGl0eS5saW5rZXJkLmNsdXN0ZXIubG9j
    YWwwWTATBgcqhkjOPQIBBggqhkjOPQMBBwNCAAQ0e7IPBlVZ03TL8UVlODllbh8b
    2pcM5mbtSGgpX9z0l3n5M70oHn715xu2szh63oBjPl2ZfOA5Bd43cJIksONQo0Iw
    QDAOBgNVHQ8BAf8EBAMCAQYwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMC
    MA8GA1UdEwEB/wQFMAMBAf8wCgYIKoZIzj0EAwIDSQAwRgIhAI7Sy8P+3TYCJBlK
    pIJSZD4lGTUyXPD4Chl/FwWdFfvyAiEA6AgCPbNCx1dOZ8RpjsN2icMRA8vwPtTx
    oSfEG/rBb68=
    -----END CERTIFICATE-----
heartbeatSchedule: '42 23 * * * '
identity:
  issuer:
    crtExpiry: "2021-08-25T23:32:17Z"
    tls:
      crtPEM: |
        -----BEGIN CERTIFICATE-----
        MIIBhDCCASmgAwIBAgIBATAKBggqhkjOPQQDAjApMScwJQYDVQQDEx5pZGVudGl0
        eS5saW5rZXJkLmNsdXN0ZXIubG9jYWwwHhcNMjAwODI1MjMzMTU3WhcNMjEwODI1
        MjMzMjE3WjApMScwJQYDVQQDEx5pZGVudGl0eS5saW5rZXJkLmNsdXN0ZXIubG9j
        YWwwWTATBgcqhkjOPQIBBggqhkjOPQMBBwNCAAQ0e7IPBlVZ03TL8UVlODllbh8b
        2pcM5mbtSGgpX9z0l3n5M70oHn715xu2szh63oBjPl2ZfOA5Bd43cJIksONQo0Iw
        QDAOBgNVHQ8BAf8EBAMCAQYwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMC
        MA8GA1UdEwEB/wQFMAMBAf8wCgYIKoZIzj0EAwIDSQAwRgIhAI7Sy8P+3TYCJBlK
        pIJSZD4lGTUyXPD4Chl/FwWdFfvyAiEA6AgCPbNCx1dOZ8RpjsN2icMRA8vwPtTx
        oSfEG/rBb68=
        -----END CERTIFICATE-----
      keyPEM: |
        -----BEGIN EC PRIVATE KEY-----
        MHcCAQEEIJaqjoDnqkKSsTqJMGeo3/1VMfJTBsMEuMWYzdJVxIhToAoGCCqGSM49
        AwEHoUQDQgAENHuyDwZVWdN0y/FFZTg5ZW4fG9qXDOZm7UhoKV/c9Jd5+TO9KB5+
        9ecbtrM4et6AYz5dmXzgOQXeN3CSJLDjUA==
        -----END EC PRIVATE KEY-----
```

Signed-off-by: Alex Leong <alex@buoyant.io>
2020-09-29 08:01:36 -07:00
Lutz Behnke de098cd52d
make api service secrets compatible to cert manager (#4737)
Currently the secrets for the proxy-injector, sp-validator webhooks and tap API service are using the Opaque secret type and linkerd-specific field names. This makes it impossible to use cert-manager (https://github.com/jetstack/cert-manager) to provisions and rotate the secrets for these services. This change converts the secrets defined in the linkerd2 helm charts and the controller use the kubernetes.io/tls format instead. This format is used for secrets containing the generated secrets by cert-manager.

Signed-off-by: Lutz Behnke <lutz.behnke@finleap.com>
2020-09-29 09:17:09 -05:00
Tarun Pothulapati d0caaa86c4
Bump k8s client-go to v0.19.2 (#5002)
Fixes #4191 #4993

This bumps Kubernetes client-go to the latest v0.19.2 (We had to switch directly to 1.19 because of this issue). Bumping to v0.19.2 required upgrading to smi-sdk-go v0.4.1. This also depends on linkerd/stern#5

This consists of the following changes:

- Fix ./bin/update-codegen.sh by adding the template path to the gen commands, as it is needed after we moved to GOMOD.
- Bump all k8s related dependencies to v0.19.2
- Generate CRD types, client code using the latest k8s.io/code-generator
- Use context.Context as the first argument, in all code paths that touch the k8s client-go interface

Signed-off-by: Tarun Pothulapati <tarunpothulapati@outlook.com>
2020-09-28 12:45:18 -05:00
Alejandro Pedraza e8f0724a71
Multicluster integration test (#4998)
This implements the run_multicluster_test() function in bin/_test-helpers.sh.

The idea is to create two clusters (source and target) using k3d, with linkerd and multicluster support in both, plus emojivoto (without vote-bot) in target, and vote-bot in source.
We then link the clusters and make sure traffic is flowing.

Detailed sequence:

Create certficates.
Install linkerd along with multicluster support in the target cluster.
Run the target1 test: install emojivoto in the target cluster (without vote-bot).
Run linkerd mc link on the target cluster.
Install linkerd along with multicluster support in the source cluster.
Apply the link resource in the source cluster.
Run the source test: Check linkerd mc gateways returns the target cluster link, and only install emojivoto's vote-bot in the source cluster. Note vote-bot's yaml defines the web-svc service as web-svc-target.emojivoto:80
Run the target2 test: Make sure web-svc in the target cluster is receiving requests.
2020-09-26 05:26:23 -05:00
Alejandro Pedraza b50ae6290d
Add support for k3d in integration tests (#4994)
* Add support for k3d in integration tests

KinD doesn't support setting LoadBalancer services out of the box. It can be added with some additional work, but it seems the solutions are not cross-platform.

K3d on the other hand facilitates this, so we'll be using k3d clusters for the multicluster integration test.

The current change sets the ground by generalizing some of the integration tests operations that were hard-coded to KinD.

- Added `bin/k3d` to wrap the setup and running of a pinned version of `k3d`.
- Refactored `bin/_test-helpers.sh` to account for tests to be run in either KinD or k3d.
- Renamed `bin/kind-load` to `bin/image-load` and make it more generic to load images for both KinD (default) and k3d. Also got rid of the no longer used `--images-host` option.
- Added a placeholder for the new `multicluster` test in the lists in `bin/_test-helpers.sh`. It starts by setting up two k3d clusters.

* Refactor handling of the `--multicluster` flag in integration tests (#4995)

Followup to #4994, based off of that branch (`alpeb/k3d-tests`).
This is more preliminary work previous to the more complete multicluster integration test.

- Removed the `--multicluster` flag from all the tests we had in `bin/_test-helpers.sh`, so only the new "multicluster" integration test will make use of that. Also got rid of the `TestUninstallMulticluster()` test in `install_test.go` to keep the multicluster stuff around, needed for the more complete multicluster test that will be implemented in a followup PR.
- Added "multicluster" to the list of tests in the `kind_integration.yml` workflow.
- For now, this new "multicluster" test in `run_multicluster_test()` is just running the install tests (`test/integration/install_test.go`) with the `--multicluster` flag.

Co-authored-by: Kevin Leimkuhler <kevin@kleimkuhler.com>
2020-09-25 16:33:17 -05:00
Kevin Leimkuhler 2ec5245d67
Add configuration for opaque ports (#4972)
## Motivation

Closes #4950

## Solution

Add the `config.linkerd.io/opaque-ports` annotation to either a namespace or pod
spec to set the proxy `LINKERD2_PROXY_INBOUND_PORTS_DISABLE_PROTOCOL_DETECTION`
environment variable.

Currently this environment variable is not used by the proxy, but will be
addressed by #4938.

## Valid values

Ports: `config.linkerd.io/opaque-ports: 4322,3306`

Port ranges: `config.linkerd.io/opaque-ports: 4320-4325`

Mixed ports and port ranges: `config.linkerd.io/opaque-ports: 4320-4325`

If the pod has named ports such as:

```
- name: nginx
  image: nginx:latest
  ports:
  - name: nginx-port
    containerPort: 80
    protocol: TCP
```

The name can also be used as a value: `config.linkerd.io/opaque-ports:
nginx-port`

Signed-off-by: Kevin Leimkuhler <kevin@kleimkuhler.com>
2020-09-25 15:36:12 -04:00
Alejandro Pedraza 0f869f2e50
Ability for int tests to use external certs generated with openssl (#4997)
Adds bin/certs-openssl, which creates self-signed root cert/key and issuer cert/key using openssl. This will be used in the two clusters set up in the multicluster integration test (followup PR), given CI already has openssl and to avoid having to install step.
Adds a new flag `--certs-path` to the integration tests, pointing to the path where those certs (ca.crt, ca.key, issuer.key and issuer.crt) will be located to be fed into linkerd install's `--identity-*` flags.
2020-09-25 11:25:29 -05:00
Alejandro Pedraza b30d35f46a
Reset service-mirror component when target's k8s API is unreachable (#4996)
When the service-mirror component can't reach the target's k8s API, the goroutine blocks and it can't be unblocked.

This was happenining specifically in the case of the multicluster integration test (still to be pushed), where the source and target clusters are created in quick succession and the target's API service doesn't always have time to be exposed before being requested by the service mirror.

The fix consists on no longer have restartClusterWatcher be side-effecting, and instead return an error. If such error is not nil then the link watcher is stopped and reset after 10 seconds.
2020-09-25 11:00:28 -05:00
Kevin Leimkuhler 55dd49e826
Add changes for edge-20.9.4 (#5004)
## edge-20.9.4

This edge release introduces support for authenticated docker registries and
fixes a recent multicluster regression.

* Fixed a regression in multicluster gateway configurations that would forbid
  inbound gateway traffic
* Upgraded bundled Grafana to v7.1.5
* Enabled Jaeger receiver in collector configuration in Helm chart (thanks
  @olivierboudet!)
* Fixed skip port configuration being skipped in CNI plugin
* Introduced support for authenticated docker registries (thanks @c-n-c!)

Signed-off-by: Kevin Leimkuhler <kevin@kleimkuhler.com>
2020-09-24 19:29:20 -04:00
Tarun Pothulapati 3d900ccc19
Integration test for smi-metrics (#4844)
* Integration test for smi-metrics

This PR adds an integration test which installs SMI-Metrics and performs
queries and matches the reply with a regex query.

Currently, We store the SMI Helm pkg locally and run the test on top, so 
That our CI does not break and we will periodically update the package
based on the newer releases of SMI-Metrics

Signed-off-by: Tarun Pothulapati <tarunpothulapati@outlook.com>
2020-09-23 22:49:20 +05:30
Tarun Pothulapati ecce5b91f6
tests: Add Calico CNI deep integration tests (#4952)
* tests: Add new CNI deep integration tests

Fixes #3944

This PR adds a new test, called cni-calico-deep which installs the Linkerd CNI
plugin on top of a cluster with Calico and performs the current integration tests on top, thus
validating various Linkerd features when CNI is enabled. For Calico
to work, special config is required for kind which is at `cni-calico.yaml`

This is different from the CNI integration tests that we run in
cloud integration which performs the CNI level integration tests.

Signed-off-by: Tarun Pothulapati <tarunpothulapati@outlook.com>
2020-09-23 19:58:28 +05:30
Nil 69ca673682
Introduce support for authenticated docker registries using imagePullSecrets, Fixes #4413 (#4898)
* Introduce support for authenticated docker registries using imagePullSecrets

Problem: Private Docker Registries are not supported for the moment as detailed in issue #4413

Solution: Every Service Account of linkerd subcomponents are Attached with imagePullSecrets,
which in turn can then pulls the docker images from authenticated private registries using them.
The imagePullSecret is configured in global.imagePullSecret parameter of values.yaml like

imagePullSecret:
  - name: <name-of-private-registry-secret-resource>

Fixes #4413

Signed-off-by: Nilakhya <nilakhya@hotmail.com>
2020-09-23 08:49:35 -05:00
Tarun Pothulapati c328de902b
CNI: Use skip ports configuration in CNI (#4974)
* CNI: Use skip ports configuration in CNI

This PR updates the install and `cmdAdd` workflow (which is called
for each new Pod creation) to retrieve and set the configured Skip
Ports. This also updates the `cmdAdd` workflow to check if the new
pod is a control plane Pod, and adds `443` to OutBoundSkipPort so
that 443 (used with k8s API) is skipped as it was causing errors because 
a resolve lookup was happening for them which is not intended.
2020-09-23 13:00:22 +05:30
OlivierB f599bf9b10
Helm chart - linkerd2-collector : enable jaeger receiver (#4783)
Fixes #4778

Signed-off-by: Olivier Boudet <o.boudet@gmail.com>
2020-09-21 12:17:04 -07:00
Zahari Dichev 0b649e3ed7
Remove double slash (#4985)
Signed-off-by: Zahari Dichev <zaharidichev@gmail.com>
2020-09-21 12:15:54 -07:00
Alejandro Pedraza 5804c15e6e
Address dependabot alert for node-fetch (#4967)
Addresses [this
alert](https://github.com/linkerd/linkerd2/network/alert/web/app/yarn.lock/node-fetch/closed)

by upgrading lingui from v2.9.1 to v2.9.2 which drops the dependency on
OpenCollective, which was the only component depending on node-fetch.

I verified the i18n tools are still working as expected.
2020-09-21 09:13:16 -05:00
Alejandro Pedraza 5eb890e735
Upgrade Grafana to v7.1.5 to get CVE fixes (#4981)
Fixes #4884

Upgrades the underlying Alpine base distro, which resolves
[CVE-2020-12723](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-12723)
and
[CVE-2020-13777](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-13777)

I tested Grafana continues to work as expected.
2020-09-21 09:12:42 -05:00
Alejandro Pedraza d6bcd1e906
Only run the ARM integration tests for stable releases (#4986) 2020-09-21 09:12:00 -05:00
Oliver Gould d98c11b47e
proxy: v2.110.0 (#4987)
This release fixes a recent regression in multicluster gateway
configurations that would forbid inbound gateway traffic. It also fixes
URI normalization for orig-proto-upgrade requests that do not include a
`Host` header.

---

* http: Simplify stacks and target types (linkerd/linkerd2-proxy#656)
* Make SkipDetect more generic as stack::MakeSwitch (linkerd/linkerd2-proxy#657)
* introduce tests for isolated services (linkerd/linkerd2-proxy#655)
* http: Put normalize_uri back on the stack (linkerd/linkerd2-proxy#659)
* inbound: Apply loop detection on the connect stack (linkerd/linkerd2-proxy#660)
* tracing: Elide redundant info in tracing contexts (linkerd/linkerd2-proxy#661)
* outbound: Reorganize outbound stacks (linkerd/linkerd2-proxy#662)
* app: Decouple stacks from listeners (linkerd/linkerd2-proxy#663)
* inbound: Split HTTP detection stack from TLS (linkerd/linkerd2-proxy#664)
* integration: Bundle tests in src (linkerd/linkerd2-proxy#665)
2020-09-18 17:36:32 -07:00
Alejandro Pedraza 51100606ca
Delete multicluster resources in `bin/test-cleanup` (#4983)
When some test failed in the middle of the
`./tests/integration/install_test.go` suite, multicluster resources can
be left-over, which `./bin/test-cleanup` wasn't removing.

This was affecting the ARM integration tests, that require good cleanup
since they use a non-transient cluster.
2020-09-18 07:38:46 -05:00
Eliza Weisman 77a55be765
update changelog for edge-20.9.3 (#4982)
This edge release includes fixes and updates for the control plane and
CLI.

* Added `--dest-cni-bin-dir` flag to the `linkerd install-cni` command,
  to configure the directory on the host where the CNI binary will be
  placed
* Removed `collector.name` and `jaeger.name` config fields from the
  tracing addon
* Updated Jaeger to 1.19.2
* Fixed a warning about deprecated Go packages in controller container
  logs
2020-09-17 12:15:35 -07:00
Alejandro Pedraza 7ab382c658
Fix Dependabot alert for `node-forge` (#4978)
Override the `node-forge` version requested by `webpack-dev-server/selfsigned` to `>=0.10.0` in order to address [CVE-2020-7720](https://github.com/advisories/GHSA-92xj-mqp7-vmcj).
2020-09-17 09:13:10 -05:00