mirror of https://github.com/linkerd/linkerd2.git
d9e9013cd9
Another attempt at fixing #6511 Even after #6524, we continued experiencing discrepancies on the linkerd-edges integration test. The problem ended up being the external prometheus instance not being injected. The injector logs revealed this: ```console 2021-07-29T13:57:10.2497460Z time="2021-07-29T13:54:15Z" level=info msg="caches synced" 2021-07-29T13:57:10.2498191Z time="2021-07-29T13:54:15Z" level=info msg="starting admin server on :9995" 2021-07-29T13:57:10.2498935Z time="2021-07-29T13:54:15Z" level=info msg="listening at :8443" 2021-07-29T13:57:10.2499945Z time="2021-07-29T13:54:18Z" level=info msg="received admission review request 2b7b4970-db40-4bda-895b-bb2e95e98265" 2021-07-29T13:57:10.2511751Z time="2021-07-29T13:54:18Z" level=debug msg="admission request: &AdmissionRequest{UID:2b7b4970-db40-4bda-895b-bb2e95e98265,Kind:/v1, Kind=Service,Resource:{ v1 services},SubResource:,Name:metrics-api,Namespace:linkerd-viz... ``` Usually one expects the webhook server to start first ("listening at :8443") and then the admin server, but in this case it happened the other way around. The admin server serves the readiness probe, so k8s was signaled that the injector was ready before it could listen to webhook requests, and given the WebhookFailurePolicy is Ignore by default, sometimes this was causing for the prometheus pod creation event to get missed, and we see in the log above that it starts by processing the pods that are created afterwards, which are the viz ones. In this fix we start first the webhook server, then block on the syncing of the k8s API, which should give enough time for the webhook to be up, and finally we start the admin server. |
||
|---|---|---|
| .. | ||
| api | ||
| cmd | ||
| gen | ||
| heartbeat | ||
| identity | ||
| k8s | ||
| proxy-injector | ||
| script/destination-client | ||
| sp-validator | ||
| webhook | ||
| Dockerfile | ||