istio.io/content/help/ops/setup/injection/index.md

title	description	weight
Sidecar Injection Webhook	Describes Istio's use of Kubernetes webhooks for automatic sidecar injection.	30

Automatic sidecar injection adds the sidecar proxy into user-created pods. It uses a MutatingWebhook to append the sidecar’s containers and volumes to each pod’s template spec during creation time. Injection can be scoped to particular sets of namespaces using the webhooks namespaceSelector mechanism. Injection can also be enabled and disabled per-pod with an annotation.

Whether or not a sidecar is injected is depends on three things:

• webhooks namespaceSelector
• default policy
• per-pod override annotation

The following truth table shows the final injection status based on these three variables.

namespaceSelector match	default policy	Pod override annotation sidecar.istio.io/inject	Sidecar injected?
yes	enabled	true	yes
yes	enabled	false	no
yes	enabled		yes
yes	disabled	true	yes
yes	disabled	false	no
yes	disabled		no
no	enabled	true	no
no	enabled	false	no
no	enabled		no
no	disabled	true	no
no	disabled	false	no
no	disabled		no

The result of sidecar injection was not what I expected

This includes an injected sidecar when it wasn't expected and a lack of injected sidecar when it was.

1. Check the webhook's namespaceSelector to determine whether the webhook is scoped to opt-in or opt-out for the target namespace.

   The namespaceSelector for opt-in will look like the following:

   {{< text bash yaml >}} $ kubectl get mutatingwebhookconfiguration istio-sidecar-injector -o yaml | grep "namespaceSelector:" -A5 namespaceSelector: matchLabels: istio-injection: enabled rules:

   • apiGroups:
     • "" {{< /text >}}

   The injection webhook will be invoked for pods created in namespaces with the istio-injection=enabled label.

   {{< text bash >}} $ kubectl get namespace -L istio-injection NAME STATUS AGE ISTIO-INJECTION default Active 18d enabled istio-system Active 3d kube-public Active 18d kube-system Active 18d {{< /text >}}

   The namespaceSelector for opt-out will look like the following:

   {{< text bash >}} $ kubectl get mutatingwebhookconfiguration istio-sidecar-injector -o yaml | grep "namespaceSelector:" -A5 namespaceSelector: matchExpressions: - key: istio-injection operator: NotIn values: - disabled rules:

   • apiGroups:
     • "" {{< /text >}}

   The injection webhook will be invoked for pods created in namespaces without the istio-injection=disabled label.

   {{< text bash >}} $ kubectl get namespace -L istio-injection NAME STATUS AGE ISTIO-INJECTION default Active 18d istio-system Active 3d disabled kube-public Active 18d disabled kube-system Active 18d disabled {{< /text >}}

   Verify the application pod's namespace is labeled properly and (re) label accordingly, e.g.

   {{< text bash >}} $ kubectl label namespace istio-system istio-injection=disabled --overwrite {{< /text >}}

   (repeat for all namespaces in which the injection webhook should be invoked for new pods)

   {{< text bash >}} $ kubectl label namespace default istio-injection=enabled --overwrite {{< /text >}}

2. Check default policy

   Check the default injection policy in the istio-sidecar-injector configmap.

   {{< text bash yaml >}} $ kubectl -n istio-system get configmap istio-sidecar-injector -o jsonpath='{.data.config}' | head policy: enabled template: |- initContainers:

   • name: istio-init image: "docker.io/jasonayoung/proxy_init:d49fa0a7f7d17f25552ad749d23f8ac73596e0cc" args:
     • "-p"
     • .MeshConfig.ProxyListenPort
     • "-u"
     • 1337 {{< /text >}}

   Allowed policy values are disabled and enabled. The default policy only applies if the webhook’s namespaceSelector matches the target namespace. Unrecognized policy values default to disabled.

3. Check the per-pod override annotation

   The default policy can be overridden with the sidecar.istio.io/inject annotation in the pod template spec’s metadata. The deployment’s metadata is ignored. Annotation value of true forces the sidecar to be injected while a value of false forces the sidecar to not be injected.

   The following annotation overrides whatever the default policy was to force the sidecar to be injected:

   {{< text bash yaml >}} $ kubectl get deployment sleep -o yaml | grep "sidecar.istio.io/inject:" -C3 template: metadata: annotations: sidecar.istio.io/inject: "true" labels: app: sleep {{< /text >}}

Pods cannot be created at all

Run kubectl describe -n namespace deployment name on the failing pod's deployment. Failure to invoke the injection webhook will typically be captured in the event log.

x509 certificate related errors

{{< text plain >}} Warning FailedCreate 3m (x17 over 8m) replicaset-controller Error creating: Internal error occurred:
failed calling admission webhook "sidecar-injector.istio.io": Post https://istio-sidecar-injector.istio-system.svc:443/inject:
x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying
to verify candidate authority certificate "Kubernetes.cluster.local") {{< /text >}}

x509: certificate signed by unknown authority errors are typically caused by an empty caBundle in the webhook configuration.

Verify the caBundle in the mutatingwebhookconfiguration matches the root certificate mounted in the istio-sidecar-injector pod.

{{< text bash >}} $ kubectl get mutatingwebhookconfiguration istio-sidecar-injector -o yaml -o jsonpath='{.webhooks[0].clientConfig.caBundle}' | md5sum 4b95d2ba22ce8971c7c92084da31faf0 - $ kubectl -n istio-system get secret istio.istio-sidecar-injector-service-account -o jsonpath='{.data.root-cert.pem}' | md5sum 4b95d2ba22ce8971c7c92084da31faf0 - {{< /text >}}

The CA certificate should match. If they do not, restart the sidecar-injector pods.

{{< text bash >}} $ kubectl -n istio-system patch deployment istio-sidecar-injector
-p "{"spec":{"template":{"metadata":{"labels":{"date":"date +'%s'"}}}}}" deployment.extensions "istio-sidecar-injector" patched {{< /text >}}

no such hosts or no endpoints available errors in deployment status

Injection is fail-close. If the istio-sidecar-injector pod is not ready, pods cannot be created. In such cases you’ll see an error about no such host (Kubernetes 1.9) or no endpoints available (>=1.10).

Kubernetes 1.9:

{{< text plain >}} Internal error occurred: failed calling admission webhook "istio-sidecar-injector.istio.io":
Post https://istio-sidecar-injector.istio-system.svc:443/admitPilot: dial tcp: lookup
istio-sidecar-injector.istio-system.svc on 169.254.169.254:53: no such host {{< /text >}}

Kubernetes 1.10:

{{< text plain >}} Internal error occurred: failed calling admission webhook "istio-sidecar-injector.istio.io":
Post https://istio-sidecar-injector.istio-system.svc:443/admitPilot?timeout=30s:
no endpoints available for service "istio-sidecar-injector" {{< /text >}}

{{< text bash >}} $ kubectl -n istio-system get pod -listio=sidecar-injector NAME READY STATUS RESTARTS AGE istio-sidecar-injector-5dbbbdb746-d676g 1/1 Running 0 2d {{< /text >}}

{{< text bash >}} $ kubectl -n istio-system get endpoints istio-sidecar-injector NAME ENDPOINTS AGE istio-sidecar-injector 10.48.6.108:10514,10.48.6.108:443 3d {{< /text >}}

If the pods or endpoints aren't ready, check the pod logs and status for any indication about why the webhook pod is failing to start and serve traffic.

{{< text bash >}} for pod in(kubectl -n istio-system get pod -listio=sidecar-injector -o jsonpath='{.items[*].metadata.name}'); do
kubectl -n istio-system logs ${pod}
done

for pod in(kubectl -n istio-system get pod -listio=sidecar-injector -o name); do
kubectl -n istio-system describe ${pod}
done {{< /text >}}