2 Commits

Author	SHA1	Message	Date
Alejandro Pedraza	578d4a19e9	Have the tap APIServer refresh its cert automatically (#5388) ... Followup to #5282, fixes #5272 in its totality. This follows the same pattern as the injector/sp-validator webhooks, leveraging `FsCredsWatcher` to watch for changes in the cert files. To reuse code from the webhooks, we moved `updateCert()` to `creds_watcher.go`, and `run()` as well (which now is called `ProcessEvents()`). The `TestNewAPIServer` test in `apiserver_test.go` was removed as it really was just testing two things: (1) that `apiServerAuth` doesn't error which is already covered in the following test, and (2) that the golib call `net.Listen("tcp", addr)` doesn't error, which we're not interested in testing here. ## How to test To test that the injector/sp-validator functionality is still correct, you can refer to #5282 The steps below are similar, but focused towards the tap component: ```bash # Create some root cert $ step certificate create linkerd-tap.linkerd.svc ca.crt ca.key --profile root-ca --no-password --insecure # configure tap's caBundle to be that root cert $ cat > linkerd-overrides.yml << EOF tap: externalSecret: true caBundle: | < ca.crt contents> EOF # Install linkerd $ bin/linkerd install --config linkerd-overrides.yml | k apply -f - # Generate an intermediatery cert with short lifespan $ step certificate create linkerd-tap.linkerd.svc ca-int.crt ca-int.key --ca ca.crt --ca-key ca.key --profile intermediate-ca --not-after 4m --no-password --insecure --san linkerd-tap.linkerd.svc # Create the secret using that intermediate cert $ kubectl create secret tls \ linkerd-tap-k8s-tls \ --cert=ca-int.crt \ --key=ca-int.key \ --namespace=linkerd # Rollout the tap pod for it to pick the new secret $ k -n linkerd rollout restart deploy/linkerd-tap # Tap should work $ bin/linkerd tap -n linkerd deploy/linkerd-web req id=0:0 proxy=in src=10.42.0.15:33040 dst=10.42.0.11:9994 tls=true :method=GET :authority=10.42.0.11:9994 :path=/metrics rsp id=0:0 proxy=in src=10.42.0.15:33040 dst=10.42.0.11:9994 tls=true :status=200 latency=1779µs end id=0:0 proxy=in src=10.42.0.15:33040 dst=10.42.0.11:9994 tls=true duration=65µs response-length=1709B # Wait 5 minutes and rollout tap again $ k -n linkerd rollout restart deploy/linkerd-tap # You'll see in the logs that the cert expired: $ k -n linkerd logs -f deploy/linkerd-tap tap 2020/12/15 16:03:41 http: TLS handshake error from 127.0.0.1:45866: remote error: tls: bad certificate 2020/12/15 16:03:41 http: TLS handshake error from 127.0.0.1:45870: remote error: tls: bad certificate # Recreate the secret $ step certificate create linkerd-tap.linkerd.svc ca-int.crt ca-int.key --ca ca.crt --ca-key ca.key --profile intermediate-ca --not-after 4m --no-password --insecure --san linkerd-tap.linkerd.svc $ k -n linkerd delete secret linkerd-tap-k8s-tls $ kubectl create secret tls \ linkerd-tap-k8s-tls \ --cert=ca-int.crt \ --key=ca-int.key \ --namespace=linkerd # Wait a few moments and you'll see the certs got reloaded and tap is working again time="2020-12-15T16:03:42Z" level=info msg="Updated certificate" addr=":8089" component=apiserver ```	2020-12-16 17:46:14 -05:00
Alejandro Pedraza	4c634a3816	Have webhooks refresh their certs automatically (#5282) ... * Have webhooks refresh their certs automatically Fixes partially #5272 In 2.9 we introduced the ability for providing the certs for `proxy-injector` and `sp-validator` through some external means like cert-manager, through the new helm setting `externalSecret`. We forgot however to have those services watch changes in their secrets, so whenever they were rotated they would fail with a cert error, with the only workaround being to restart those pods to pick the new secrets. This addresses that by first abstracting out `FsCredsWatcher` from the identity controller, which now lives under `pkg/tls`. The webhook's logic in `launcher.go` no longer reads the certs before starting the https server, moving that instead into `server.go` which in a similar way as identity will receive events from `FsCredsWatcher` and update `Server.cert`. We're leveraging `http.Server.TLSConfig.GetCertificate` which allows us to provide a function that will return the current cert for every incoming request. ### How to test ```bash # Create some root cert $ step certificate create linkerd-proxy-injector.linkerd.svc ca.crt ca.key \ --profile root-ca --no-password --insecure --san linkerd-proxy-injector.linkerd.svc # configure injector's caBundle to be that root cert $ cat > linkerd-overrides.yaml << EOF proxyInjector: externalSecret: true caBundle: | < ca.crt contents> EOF # Install linkerd. The injector won't start untill we create the secret below $ bin/linkerd install --controller-log-level debug --config linkerd-overrides.yaml | k apply -f - # Generate an intermediatery cert with short lifespan step certificate create linkerd-proxy-injector.linkerd.svc ca-int.crt ca-int.key --ca ca.crt --ca-key ca.key --profile intermediate-ca --not-after 4m --no-password --insecure --san linkerd-proxy-injector.linkerd.svc # Create the secret using that intermediate cert $ kubectl create secret tls \ linkerd-proxy-injector-k8s-tls \ --cert=ca-int.crt \ --key=ca-int.key \ --namespace=linkerd # start following the injector log $ k -n linkerd logs -f -l linkerd.io/control-plane-component=proxy-injector -c proxy-injector # Inject emojivoto. The pods should be injected normally $ bin/linkerd inject https://run.linkerd.io/emojivoto.yml | kubectl apply -f - # Wait about 5 minutes and delete a pod $ k -n emojivoto delete po -l app=emoji-svc # You'll see it won't be injected, and something like "remote error: tls: bad certificate" will appear in the injector logs. # Regenerate the intermediate cert $ step certificate create linkerd-proxy-injector.linkerd.svc ca-int.crt ca-int.key --ca ca.crt --ca-key ca.key --profile intermediate-ca --not-after 4m --no-password --insecure --san linkerd-proxy-injector.linkerd.svc # Delete the secret and recreate it $ k -n linkerd delete secret linkerd-proxy-injector-k8s-tls $ kubectl create secret tls \ linkerd-proxy-injector-k8s-tls \ --cert=ca-int.crt \ --key=ca-int.key \ --namespace=linkerd # Wait a couple of minutes and you'll see some filesystem events in the injector log along with a "Certificate has been updated" entry # Then delete the pod again and you'll see it gets injected this time $ k -n emojivoto delete po -l app=emoji-svc ```	2020-12-04 16:25:59 -05:00