mirror of https://github.com/istio/istio.io.git
259 lines
10 KiB
Markdown
259 lines
10 KiB
Markdown
---
|
||
title: Sidecar Injection Problems
|
||
description: Resolve common problems with Istio's use of Kubernetes webhooks for automatic sidecar injection.
|
||
force_inline_toc: true
|
||
weight: 40
|
||
aliases:
|
||
- /docs/ops/troubleshooting/injection
|
||
owner: istio/wg-user-experience-maintainers
|
||
test: n/a
|
||
---
|
||
|
||
## The result of sidecar injection was not what I expected
|
||
|
||
This includes an injected sidecar when it wasn't expected and a lack
|
||
of injected sidecar when it was.
|
||
|
||
1. Ensure your pod is not in the `kube-system` or `kube-public` namespace.
|
||
Automatic sidecar injection will be ignored for pods in these namespaces.
|
||
|
||
1. Ensure your pod does not have `hostNetwork: true` in its pod spec.
|
||
Automatic sidecar injection will be ignored for pods that are on the host network.
|
||
|
||
The sidecar model assumes that the iptables changes required for Envoy to intercept
|
||
traffic are within the pod. For pods on the host network this assumption is violated,
|
||
and this can lead to routing failures at the host level.
|
||
|
||
1. Check the webhook's `namespaceSelector` to determine whether the
|
||
webhook is scoped to opt-in or opt-out for the target namespace.
|
||
|
||
The `namespaceSelector` for opt-in will look like the following:
|
||
|
||
{{< text bash yaml >}}
|
||
$ kubectl get mutatingwebhookconfiguration istio-sidecar-injector -o yaml | grep "namespaceSelector:" -A5
|
||
namespaceSelector:
|
||
matchLabels:
|
||
istio-injection: enabled
|
||
rules:
|
||
- apiGroups:
|
||
- ""
|
||
{{< /text >}}
|
||
|
||
The injection webhook will be invoked for pods created
|
||
in namespaces with the `istio-injection=enabled` label.
|
||
|
||
{{< text bash >}}
|
||
$ kubectl get namespace -L istio-injection
|
||
NAME STATUS AGE ISTIO-INJECTION
|
||
default Active 18d enabled
|
||
istio-system Active 3d
|
||
kube-public Active 18d
|
||
kube-system Active 18d
|
||
{{< /text >}}
|
||
|
||
The `namespaceSelector` for opt-out will look like the following:
|
||
|
||
{{< text bash >}}
|
||
$ kubectl get mutatingwebhookconfiguration istio-sidecar-injector -o yaml | grep "namespaceSelector:" -A5
|
||
namespaceSelector:
|
||
matchExpressions:
|
||
- key: istio-injection
|
||
operator: NotIn
|
||
values:
|
||
- disabled
|
||
rules:
|
||
- apiGroups:
|
||
- ""
|
||
{{< /text >}}
|
||
|
||
The injection webhook will be invoked for pods created in namespaces
|
||
without the `istio-injection=disabled` label.
|
||
|
||
{{< text bash >}}
|
||
$ kubectl get namespace -L istio-injection
|
||
NAME STATUS AGE ISTIO-INJECTION
|
||
default Active 18d
|
||
istio-system Active 3d disabled
|
||
kube-public Active 18d disabled
|
||
kube-system Active 18d disabled
|
||
{{< /text >}}
|
||
|
||
Verify the application pod's namespace is labeled properly and (re) label accordingly, e.g.
|
||
|
||
{{< text bash >}}
|
||
$ kubectl label namespace istio-system istio-injection=disabled --overwrite
|
||
{{< /text >}}
|
||
|
||
(repeat for all namespaces in which the injection webhook should be invoked for new pods)
|
||
|
||
{{< text bash >}}
|
||
$ kubectl label namespace default istio-injection=enabled --overwrite
|
||
{{< /text >}}
|
||
|
||
1. Check default policy
|
||
|
||
Check the default injection policy in the `istio-sidecar-injector configmap`.
|
||
|
||
{{< text bash yaml >}}
|
||
$ kubectl -n istio-system get configmap istio-sidecar-injector -o jsonpath='{.data.config}' | grep policy:
|
||
policy: enabled
|
||
{{< /text >}}
|
||
|
||
Allowed policy values are `disabled` and `enabled`. The default policy
|
||
only applies if the webhook’s `namespaceSelector` matches the target
|
||
namespace. Unrecognized policy causes injection to be disabled completely.
|
||
|
||
1. Check the per-pod override annotation
|
||
|
||
The default policy can be overridden with the
|
||
`sidecar.istio.io/inject` label in the _pod template spec’s metadata_.
|
||
The deployment’s metadata is ignored. Label value
|
||
of `true` forces the sidecar to be injected while a value of
|
||
`false` forces the sidecar to _not_ be injected.
|
||
|
||
The following label overrides whatever the default `policy` was
|
||
to force the sidecar to be injected:
|
||
|
||
{{< text bash yaml >}}
|
||
$ kubectl get deployment sleep -o yaml | grep "sidecar.istio.io/inject:" -B4
|
||
template:
|
||
metadata:
|
||
labels:
|
||
app: sleep
|
||
sidecar.istio.io/inject: "true"
|
||
{{< /text >}}
|
||
|
||
## Pods cannot be created at all
|
||
|
||
Run `kubectl describe -n namespace deployment name` on the failing
|
||
pod's deployment. Failure to invoke the injection webhook will
|
||
typically be captured in the event log.
|
||
|
||
### x509 certificate related errors
|
||
|
||
{{< text plain >}}
|
||
Warning FailedCreate 3m (x17 over 8m) replicaset-controller Error creating: Internal error occurred: \
|
||
failed calling admission webhook "sidecar-injector.istio.io": Post https://istiod.istio-system.svc:443/inject: \
|
||
x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying \
|
||
to verify candidate authority certificate "Kubernetes.cluster.local")
|
||
{{< /text >}}
|
||
|
||
`x509: certificate signed by unknown authority` errors are typically
|
||
caused by an empty `caBundle` in the webhook configuration.
|
||
|
||
Verify the `caBundle` in the `mutatingwebhookconfiguration` matches the
|
||
root certificate mounted in the `istiod` pod.
|
||
|
||
{{< text bash >}}
|
||
$ kubectl get mutatingwebhookconfiguration istio-sidecar-injector -o yaml -o jsonpath='{.webhooks[0].clientConfig.caBundle}' | md5sum
|
||
4b95d2ba22ce8971c7c92084da31faf0 -
|
||
$ kubectl -n istio-system get configmap istio-ca-root-cert -o jsonpath='{.data.root-cert\.pem}' | base64 -w 0 | md5sum
|
||
4b95d2ba22ce8971c7c92084da31faf0 -
|
||
{{< /text >}}
|
||
|
||
The CA certificate should match. If they do not, restart the
|
||
istiod pods.
|
||
|
||
{{< text bash >}}
|
||
$ kubectl -n istio-system patch deployment istiod \
|
||
-p "{\"spec\":{\"template\":{\"metadata\":{\"labels\":{\"date\":\"`date +'%s'`\"}}}}}"
|
||
deployment.extensions "istiod" patched
|
||
{{< /text >}}
|
||
|
||
### `no such hosts` or `no endpoints available` errors in deployment status
|
||
|
||
Injection is fail-close. If the `istio-sidecar-injector` pod is not ready, pods
|
||
cannot be created. In such cases you’ll see an error about `no endpoints available`.
|
||
|
||
{{< text plain >}}
|
||
Internal error occurred: failed calling admission webhook "istio-sidecar-injector.istio.io": \
|
||
Post https://istio-sidecar-injector.istio-system.svc:443/admitPilot?timeout=30s: \
|
||
no endpoints available for service "istio-sidecar-injector"
|
||
{{< /text >}}
|
||
|
||
{{< text bash >}}
|
||
$ kubectl -n istio-system get pod -listio=sidecar-injector
|
||
NAME READY STATUS RESTARTS AGE
|
||
istio-sidecar-injector-5dbbbdb746-d676g 1/1 Running 0 2d
|
||
{{< /text >}}
|
||
|
||
{{< text bash >}}
|
||
$ kubectl -n istio-system get endpoints istio-sidecar-injector
|
||
NAME ENDPOINTS AGE
|
||
istio-sidecar-injector 10.48.6.108:15014,10.48.6.108:443 3d
|
||
{{< /text >}}
|
||
|
||
If the pods or endpoints aren't ready, check the pod logs and status
|
||
for any indication about why the webhook pod is failing to start and
|
||
serve traffic.
|
||
|
||
{{< text bash >}}
|
||
$ for pod in $(kubectl -n istio-system get pod -listio=sidecar-injector -o jsonpath='{.items[*].metadata.name}'); do \
|
||
kubectl -n istio-system logs ${pod} \
|
||
done
|
||
|
||
$ for pod in $(kubectl -n istio-system get pod -listio=sidecar-injector -o name); do \
|
||
kubectl -n istio-system describe ${pod} \
|
||
done
|
||
{{< /text >}}
|
||
|
||
## Automatic sidecar injection fails if the Kubernetes API server has proxy settings
|
||
|
||
When the Kubernetes API server includes proxy settings such as:
|
||
|
||
{{< text yaml >}}
|
||
env:
|
||
- name: http_proxy
|
||
value: http://proxy-wsa.esl.foo.com:80
|
||
- name: https_proxy
|
||
value: http://proxy-wsa.esl.foo.com:80
|
||
- name: no_proxy
|
||
value: 127.0.0.1,localhost,dockerhub.foo.com,devhub-docker.foo.com,10.84.100.125,10.84.100.126,10.84.100.127
|
||
{{< /text >}}
|
||
|
||
With these settings, Sidecar injection fails. The only related failure log can be found in `kube-apiserver` log:
|
||
|
||
{{< text plain >}}
|
||
W0227 21:51:03.156818 1 admission.go:257] Failed calling webhook, failing open sidecar-injector.istio.io: failed calling admission webhook "sidecar-injector.istio.io": Post https://istio-sidecar-injector.istio-system.svc:443/inject: Service Unavailable
|
||
{{< /text >}}
|
||
|
||
Make sure both pod and service CIDRs are not proxied according to `*_proxy` variables. Check the `kube-apiserver` files and logs to verify the configuration and whether any requests are being proxied.
|
||
|
||
One workaround is to remove the proxy settings from the `kube-apiserver` manifest, another workaround is to include `istio-sidecar-injector.istio-system.svc` or `.svc` in the `no_proxy` value. Make sure that `kube-apiserver` is restarted after each workaround.
|
||
|
||
An [issue](https://github.com/kubernetes/kubeadm/issues/666) was filed with Kubernetes related to this and has since been closed.
|
||
[https://github.com/kubernetes/kubernetes/pull/58698#discussion_r163879443](https://github.com/kubernetes/kubernetes/pull/58698#discussion_r163879443)
|
||
|
||
## Limitations for using Tcpdump in pods
|
||
|
||
Tcpdump doesn't work in the sidecar pod - the container doesn't run as root. However any other container in the same pod will see all the packets, since the
|
||
network namespace is shared. `iptables` will also see the pod-wide configuration.
|
||
|
||
Communication between Envoy and the app happens on 127.0.0.1, and is not encrypted.
|
||
|
||
## Cluster is not scaled down automatically
|
||
|
||
Due to the fact that the sidecar container mounts a local storage volume, the
|
||
node autoscaler is unable to evict nodes with the injected pods. This is
|
||
a [known issue](https://github.com/kubernetes/autoscaler/issues/3947). The workaround is
|
||
to add a pod annotation `"cluster-autoscaler.kubernetes.io/safe-to-evict":
|
||
"true"` to the injected pods.
|
||
|
||
## Pod or containers start with network issues if istio-proxy is not ready
|
||
|
||
Many applications execute commands or checks during startup, which require network connectivity. This can cause application containers to hang or restart if the `istio-proxy` sidecar container is not ready.
|
||
|
||
To avoid this, set `holdApplicationUntilProxyStarts` to `true`. This causes the sidecar injector to inject the sidecar at the start of the pod’s container list, and configures it to block the start of all other containers until the proxy is ready.
|
||
|
||
This can be added as a global config option:
|
||
|
||
{{< text yaml >}}
|
||
values.global.proxy.holdApplicationUntilProxyStarts: true
|
||
{{< /text >}}
|
||
|
||
or as a pod annotation:
|
||
|
||
{{< text yaml >}}
|
||
proxy.istio.io/config: '{ "holdApplicationUntilProxyStarts": true }'
|
||
{{< /text >}}
|