istio.io/archive/v1.3/docs/ops/troubleshooting/feed.xml

<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom"><channel><title>Troubleshooting on Istio</title><link>/v1.3/docs/ops/troubleshooting/</link><description>Recent content in Troubleshooting on Istio</description><generator>Hugo -- gohugo.io</generator><language>en-us</language><atom:link href="/v1.3/docs/ops/troubleshooting/feed.xml" rel="self" type="application/rss+xml"/><item><title>Using the istioctl command-line tool</title><link>/v1.3/docs/ops/troubleshooting/istioctl/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>/v1.3/docs/ops/troubleshooting/istioctl/</guid><description>Overview You can gain insights into what individual components are doing by inspecting their logs or peering inside via introspection. If that&amp;rsquo;s insufficient, the steps below explain how to get under the hood.
The istioctl tool is a configuration command line utility that allows service operators to debug and diagnose their Istio service mesh deployments. The Istio project also includes two helpful scripts for istioctl that enable auto-completion for Bash and ZSH.</description></item><item><title>Network Problems</title><link>/v1.3/docs/ops/troubleshooting/network-issues/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>/v1.3/docs/ops/troubleshooting/network-issues/</guid><description>Requests are rejected by Envoy Requests may be rejected for various reasons. The best way to understand why requests are being rejected is by inspecting Envoy&amp;rsquo;s access logs. By default, access logs are output to the standard output of the container. Run the following command to see the log:
$ kubectl logs PODNAME -c istio-proxy -n NAMESPACE In the default access log format, Envoy response flags and Mixer policy status are located after the response code, if you are using a custom log format, make sure to include %RESPONSE_FLAGS% and %DYNAMIC_METADATA(istio.</description></item><item><title>Security Problems</title><link>/v1.3/docs/ops/troubleshooting/security-issues/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>/v1.3/docs/ops/troubleshooting/security-issues/</guid><description>End-user Authentication Fails With Istio, you can enable authentication for end users. Currently, the end user credential supported by the Istio authentication policy is JWT. The following is a guide for troubleshooting the end user JWT authentication.
Check your Istio authentication policy, principalBinding should be set as USE_ORIGIN to authenticate the end user.
If jwksUri isn’t set, make sure the JWT issuer is of url format and url + /.</description></item><item><title>Sidecar Injection Problems</title><link>/v1.3/docs/ops/troubleshooting/injection/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>/v1.3/docs/ops/troubleshooting/injection/</guid><description>The result of sidecar injection was not what I expected This includes an injected sidecar when it wasn&amp;rsquo;t expected and a lack of injected sidecar when it was.
Ensure your pod is not in the kube-system or kube-public namespace. Automatic sidecar injection will be ignored for pods in these namespaces.
Ensure your pod does not have hostNetwork: true in its pod spec. Automatic sidecar injection will be ignored for pods that are on the host network.</description></item><item><title>Repairing Citadel</title><link>/v1.3/docs/ops/troubleshooting/repairing-citadel/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>/v1.3/docs/ops/troubleshooting/repairing-citadel/</guid><description>Citadel does not support multiple instances. Running multiple Citadel instances may introduce race conditions and lead to system outages. Workloads with new Kubernetes service accounts can not be started when Citadel is disabled for maintenance since they can&amp;rsquo;t get their certificates generated. Citadel is not a critical data plane component. The default workload certificate lifetime is 3 months. Certificates will be rotated by Citadel before they expire.</description></item><item><title>Debugging Envoy and Pilot</title><link>/v1.3/docs/ops/troubleshooting/proxy-cmd/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>/v1.3/docs/ops/troubleshooting/proxy-cmd/</guid><description>Istio provides two very valuable commands to help diagnose traffic management configuration problems, the proxy-status and proxy-config commands. The proxy-status command allows you to get an overview of your mesh and identify the proxy causing the problem. Then proxy-config can be used to inspect Envoy configuration and diagnose the issue.
If you want to try the commands described below, you can either:
Have a Kubernetes cluster with Istio and Bookinfo installed (e.</description></item><item><title>Galley Configuration Problems</title><link>/v1.3/docs/ops/troubleshooting/validation/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>/v1.3/docs/ops/troubleshooting/validation/</guid><description>Seemingly valid configuration is rejected Manually verify your configuration is correct, cross-referencing Istio API reference when necessary.
Invalid configuration is accepted Verify the istio-galley validationwebhookconfiguration exists and is correct. The apiVersion, apiGroup, and resource of the invalid configuration should be listed in one of the two webhooks entries.
$ kubectl get validatingwebhookconfiguration istio-galley -o yaml apiVersion: admissionregistration.k8s.io/v1beta1 kind: ValidatingWebhookConfiguration metadata: labels: app: istio-galley name: istio-galley ownerReferences: - apiVersion: extensions/v1beta1 blockOwnerDeletion: true controller: true kind: Deployment name: istio-galley uid: 5c64585d-91c6-11e8-a98a-42010a8001a8 webhooks: - clientConfig: # caBundle should be non-empty.</description></item><item><title>Missing Metrics</title><link>/v1.3/docs/ops/troubleshooting/missing-metrics/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>/v1.3/docs/ops/troubleshooting/missing-metrics/</guid><description>The procedures below help you diagnose problems where metrics you are expecting to see reported and not being collected.
The expected flow for metrics is:
Envoy reports attributes from requests asynchronously to Mixer in a batch.
Mixer translates the attributes into instances based on the operator-provided configuration.
Mixer hands the instances to Mixer adapters for processing and backend storage.
The backend storage systems record the metrics data.</description></item><item><title>Missing Grafana Output</title><link>/v1.3/docs/ops/troubleshooting/grafana/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>/v1.3/docs/ops/troubleshooting/grafana/</guid><description>If you&amp;rsquo;re unable to get Grafana output when connecting from a local web client to Istio remotely hosted, you should validate the client and server date and time match.
The time of the web client (e.g. Chrome) affects the output from Grafana. A simple solution to this problem is to verify a time synchronization service is running correctly within the Kubernetes cluster and the web client machine also is correctly using a time synchronization service.</description></item><item><title>Missing Zipkin Traces</title><link>/v1.3/docs/ops/troubleshooting/missing-traces/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>/v1.3/docs/ops/troubleshooting/missing-traces/</guid><description>No traces appearing in Zipkin when running Istio locally on Mac Istio is installed and everything seems to be working except there are no traces showing up in Zipkin when there should be.
This may be caused by a known Docker issue where the time inside containers may skew significantly from the time on the host machine. If this is the case, when you select a very long date range in Zipkin you will see the traces appearing as much as several days too early.</description></item><item><title>Understand your Mesh with istioctl describe</title><link>/v1.3/docs/ops/troubleshooting/istioctl-describe/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>/v1.3/docs/ops/troubleshooting/istioctl-describe/</guid><description>The following information describes an experimental feature, which is intended for evaluation purposes only. In Istio 1.3, we included the istioctl experimental describe command. This CLI command provides you with the information needed to understand the configuration impacting a pod. This guide shows you how to use this experimental sub-command to see if a pod is in the mesh and verify its configuration.
The basic usage of the command is as follows:</description></item><item><title>Component Logging</title><link>/v1.3/docs/ops/troubleshooting/component-logging/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>/v1.3/docs/ops/troubleshooting/component-logging/</guid><description>Istio components are built with a flexible logging framework which provides a number of features and controls to help operate these components and facilitate diagnostics. You control these logging features by passing command-line options when starting the components.
Logging scopes Logging messages output by a component are categorized by scopes. A scope represents a set of related log messages which you can control as a whole. Different components have different scopes, depending on the features the component provides.</description></item><item><title>Component Introspection</title><link>/v1.3/docs/ops/troubleshooting/controlz/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>/v1.3/docs/ops/troubleshooting/controlz/</guid><description>Istio components are built with a flexible introspection framework which makes it easy to inspect and manipulate the internal state of a running component. Components open a port which can be used from a web browser to get an interactive view into the state of the component, or via REST for access and control from external tools.
Mixer, Pilot, and Galley all implement the ControlZ functionality. When these components start, a message is logged indicating the IP address and port to connect to in order to interact with ControlZ.</description></item><item><title>Tcpdump Limitations</title><link>/v1.3/docs/ops/troubleshooting/tcpdump-notes/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>/v1.3/docs/ops/troubleshooting/tcpdump-notes/</guid><description>Tcpdump doesn&amp;rsquo;t work in the sidecar pod - the container doesn&amp;rsquo;t run as root. However any other container in the same pod will see all the packets, since the network namespace is shared. iptables will also see the pod-wide configuration.
Communication between Envoy and the app happens on 127.0.0.1, and is not encrypted.</description></item></channel></rss>