mirror of https://github.com/istio/istio.io.git
64 lines
21 KiB
HTML
64 lines
21 KiB
HTML
<!DOCTYPE html><html lang="en" itemscope itemtype="https://schema.org/WebPage" style="overflow-y: scroll;"><head><meta charset="utf-8"><meta http-equiv="X-UA-Compatible" content="IE=edge"><meta name="viewport" content="width=device-width, initial-scale=1"><meta name="title" content="Troubleshooting Guide"><meta name="og:title" content="Troubleshooting Guide"><meta name="og:image" content="/v0.2/img/logo.png"/><meta name="description" content="Practical advice on practical problems with Istio"><meta name="og:description" content="Practical advice on practical problems with Istio"><title>Istioldie 0.2 / Troubleshooting Guide</title><script> window.ga=window.ga||function(){(ga.q=ga.q||[]).push(arguments)};ga.l=+new Date; ga('create', 'UA-98480406-2', 'auto'); ga('send', 'pageview'); </script> <script async src='https://www.google-analytics.com/analytics.js'></script><link rel="alternate" type="application/rss+xml" title="Istio Blog RSS" href="/v0.2/feed.xml"><link rel="apple-touch-icon" href="/v0.2/favicons/apple-touch-icon.png" sizes="180x180"><link rel="icon" type="image/png" href="/v0.2/favicons/android-chrome-96x96.png" sizes="96x96" ><link rel="icon" type="image/png" href="/v0.2/favicons/favicon-32x32.png" sizes="32x32"><link rel="icon" type="image/png" href="/v0.2/favicons/favicon-16x16.png" sizes="16x16"><link rel="manifest" href="/v0.2/favicons/manifest.json"><link rel="mask-icon" href="/v0.2/favicons/safari-pinned-tab.svg" color="#2DA6B0"><meta name="msapplication-TileColor" content="#ffffff"><meta name="msapplication-TileImage" content="/v0.2/favicons/mstile-150x150.png"><link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Roboto:400,100,100italic,300,300italic,400italic,500,500italic,700,700italic,900,900italic"><link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/css/bootstrap.min.css"><link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/4.7.0/css/font-awesome.min.css"><link rel="stylesheet" href="/v0.2/css/all.css"><link rel="stylesheet" href="/v0.2/css/prism.css"> <script src="https://cdnjs.cloudflare.com/ajax/libs/jquery/2.2.4/jquery.min.js"></script></head><body class="language-unknown"><div class="nav-hero-container" style="z-index: 200000;"><nav id="header-nav" class="navbar navbar-inverse" role="navigation"><div class="container"><div class="row"><div class="col-md-11 nofloat center-block "><div class="navbar-header"> <button type="button" class="hamburger navbar-toggle collapsed" data-toggle="collapse" data-target="#navbar-collapse-1" aria-expanded="false"> <span class="sr-only">Toggle navigation</span> <span class="icon-bar"></span> <span class="icon-bar"></span> <span class="icon-bar"></span> </button> <a class="navbar-brand" href="/v0.2/"><div> <img src="/v0.2/img/logo.png" alt="Istio" width="36px" height="54px"/> <span class="brand-name">Istioldie 0.2</span></div></a></div><div class="collapse navbar-collapse" id="navbar-collapse-1"><ul class="nav navbar-nav navbar-right"><li><a href="/v0.2/about/" >About</a></li><li><a href="/v0.2/docs/" >Docs</a></li><li><a href="/v0.2/blog/" >Blog</a></li><li><a href="/v0.2/community/" >Community</a></li><li><a href="/v0.2/faq/" >FAQ</a></li><li class="dropdown"><li class="dropdown"> <a class="dropdown-toggle" data-toggle="dropdown" href=""> <i class='fa fa-lg fa-cog'></i> <span class="caret"></span> </a><ul class="dropdown-menu"><h6 class="dropdown-header">Other versions of this site</h6><li> <a href="https://istio.io">Current Release</a></li><li> <a href="https://preliminary.istio.io">Next Release</a></li><li> <a href="https://archive.istio.io">Older Releases</a></li></ul></li><li><form name="cse" id="searchbox_demo" class="navbar-form navbar-right" role="search"> <input type="hidden" name="cx" value="013699703217164175118:iwwf17ikgf4" /> <input type="hidden" name="ie" value="utf-8" /> <input type="hidden" name="hl" value="en" /><div class="form-group"><div class="input-group"> <input name="q" class="form-control" type="text" size="30" /><div class="input-group-addon"> <span class="btn-search glyphicon glyphicon-search"></span></div></div></div></form> <script type="text/javascript" src="https://www.google.com/cse/brand?form=searchbox_demo"></script></li></ul></div></div></div></div></nav></div><div class="container troubleshooting"><div class="row"><div class="col-sm-12 col-md-10 col-lg-9 nofloat center-block markdown"><div id="toc" class="toc"></div><h1 id="troubleshooting-guide">Troubleshooting Guide</h1><p>Oh no! You’re having trouble? Below is a list of solutions to common problems.</p><blockquote><p><img src="/v0.2/img/troubleshooting.svg" alt="Troubleshooting" title="Troubleshooting" style="width: 32px; display:inline" /> Hint: If you don’t find what you need here, be sure to check out the <a href="https://groups.google.com/forum/#!forum/istio-users">istio-users@</a> mailing list to see if someone else has already found a solution. If after that you’re still stuck, consider <a href="/v0.2/bugs/">reporting a bug</a>.</p></blockquote><h2 id="no-traces-appearing-in-zipkin-when-running-istio-locally-on-mac">No traces appearing in Zipkin when running Istio locally on Mac</h2><p>Istio is installed and everything seems to be working except there are no traces showing up in Zipkin when there should be.</p><p>This may be caused by a known <a href="https://github.com/docker/for-mac/issues/1260">Docker issue</a> where the time inside containers may skew significantly from the time on the host machine. If this is the case, when you select a very long date range in Zipin you will see the traces appearing as much as several days too early.</p><p>You can also confirm this problem by comparing the date inside a docker container to outside:</p><pre><code class="language-bash">$ docker run --entrypoint date gcr.io/istio-testing/ubuntu-16-04-slave:latest
|
||
Sun Jun 11 11:44:18 UTC 2017
|
||
$ date -u
|
||
Thu Jun 15 02:25:42 UTC 2017
|
||
</code></pre><p>To fix the problem, you’ll need to shutdown and then restart Docker before reinstalling Istio.</p><h2 id="envoy-wont-connect-to-my-http10-service">Envoy won’t connect to my HTTP/1.0 service</h2><p>Envoy requires HTTP/1.1 or HTTP/2 traffic for upstream services. For example, when using <a href="https://www.nginx.com/">NGINX</a> for serving traffic behind Envoy, you will need to set the <a href="http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_http_version">proxy_http_version</a> directive in your NGINX config to be “1.1”, since the NGINX default is 1.0</p><p>Example config:</p><pre><code>upstream http_backend {
|
||
server 127.0.0.1:8080;
|
||
|
||
keepalive 16;
|
||
}
|
||
|
||
server {
|
||
...
|
||
|
||
location /http/ {
|
||
proxy_pass http://http_backend;
|
||
proxy_http_version 1.1;
|
||
proxy_set_header Connection "";
|
||
...
|
||
}
|
||
}
|
||
</code></pre><h2 id="where-are-the-metrics-for-my-service">Where are the metrics for my service?</h2><p>The expected flow of metrics is:</p><ol><li>Envoy reports attributes to Mixer in batch (asynchronously from requests)</li><li>Mixer translates the attributes from Mixer into instances based on operator-provided configuration.</li><li>The instances are handed to Mixer adapters for processing and backend storage.</li><li>The backend storage systems record metrics data.</li></ol><p>The default installations of Mixer ship with a <a href="http://prometheus.io/">Prometheus</a> adapter, as well as configuration for generating a basic set of metric values and sending them to the Prometheus adapter. The <a href="/v0.2/docs/tasks/telemetry/querying-metrics.html#about-the-prometheus-add-on">Prometheus add-on</a> also supplies configuration for an instance of Prometheus to scrape Mixer for metrics.</p><p>If you do not see the expected metrics in the Istio Dashboard and/or via Prometheus queries, there may be an issue at any of the steps in the flow listed above. Below is a set of instructions to troubleshoot each of those steps.</p><h3 id="verify-mixer-is-receiving-report-calls">Verify Mixer is receiving Report calls</h3><p>Mixer generates metrics for monitoring the behavior of Mixer itself. Check these metrics.</p><ol><li><p>Establish a connection to the Mixer self-monitoring endpoint.</p><p>In Kubernetes environments, execute the following command:</p><pre><code class="language-bash">kubectl -n istio-system port-forward <mixer pod> 9093 &
|
||
</code></pre></li><li><p>Verify successful report calls.</p><p>On the <a href="http://localhost:9093/metrics">Mixer self-monitoring endpoint</a>, search for <code>grpc_server_handled_total</code>.</p><p>You should see something like:</p><pre><code>grpc_server_handled_total{grpc_code="OK",grpc_method="Report",grpc_service="istio.mixer.v1.Mixer",grpc_type="unary"} 68
|
||
</code></pre></li></ol><p>If you do not see any data for <code>grpc_server_handled_total</code> with a <code>grpc_method="Report"</code>, then Mixer is not being called by Envoy to report telemetry. In this case, ensure that the services have been properly integrated into the mesh (either by via <a href="/v0.2/docs/setup/kubernetes/sidecar-injection.html#automatic-sidecar-injection">automatic</a> or <a href="/v0.2/docs/setup/kubernetes/sidecar-injection.html#manual-sidecar-injection">manual</a> sidecar injection).</p><h3 id="verify-mixer-metrics-configuration-exists">Verify Mixer metrics configuration exists</h3><ol><li><p>Verify Mixer rules exist.</p><p>In Kubernetes environments, issue the following command:</p><pre><code class="language-bash">kubectl get rules --all-namespaces
|
||
</code></pre><p>With the default configuration, you should see something like:</p><pre><code>NAMESPACE NAME KIND
|
||
istio-system promhttp rule.v1alpha2.config.istio.io
|
||
istio-system promtcp rule.v1alpha2.config.istio.io
|
||
istio-system stdio rule.v1alpha2.config.istio.io
|
||
</code></pre><p>If you do not see anything named <code>promhttp</code> or <code>promtcp</code>, then there is no Mixer configuration for sending metric instances to a Prometheus adapter. You will need to supply configuration for rules that connect Mixer metric instances to a Prometheus handler (<a href="https://github.com/istio/istio/blob/master/install/kubernetes/istio.yaml#L892">example</a>).</p></li><li><p>Verify Prometheus handler config exists.</p><p>In Kubernetes environments, issue the following command:</p><pre><code class="language-bash">kubectl get prometheuses.config.istio.io --all-namespaces
|
||
</code></pre><p>The expected output is:</p><pre><code>NAMESPACE NAME KIND
|
||
istio-system handler prometheus.v1alpha2.config.istio.io
|
||
</code></pre><p>If there are no prometheus handlers configured, you will need to reconfigure Mixer with the appropriate handler configuration (<a href="https://github.com/istio/istio/blob/master/install/kubernetes/istio.yaml#L819">example</a>)</p></li><li><p>Verify Mixer metric instances config exists.</p><p>In Kubernetes environments, issue the following command:</p><pre><code class="language-bash">kubectl get metrics.config.istio.io --all-namespaces
|
||
</code></pre><p>The expected output is:</p><pre><code>NAMESPACE NAME KIND
|
||
istio-system requestcount metric.v1alpha2.config.istio.io
|
||
istio-system requestduration metric.v1alpha2.config.istio.io
|
||
istio-system requestsize metric.v1alpha2.config.istio.io
|
||
istio-system responsesize metric.v1alpha2.config.istio.io
|
||
istio-system stackdriverrequestcount metric.v1alpha2.config.istio.io
|
||
istio-system stackdriverrequestduration metric.v1alpha2.config.istio.io
|
||
istio-system stackdriverrequestsize metric.v1alpha2.config.istio.io
|
||
istio-system stackdriverresponsesize metric.v1alpha2.config.istio.io
|
||
istio-system tcpbytereceived metric.v1alpha2.config.istio.io
|
||
istio-system tcpbytesent metric.v1alpha2.config.istio.io
|
||
</code></pre><p>If there are no metric instances configured, you will need to reconfigure Mixer with the appropriate instance configuration (<a href="https://github.com/istio/istio/blob/master/install/kubernetes/istio.yaml#L727">example</a>)</p></li><li><p>Verify Mixer configuration resolution is working for your service.</p><ol><li><p>Establish a connection to the Mixer self-monitoring endpoint.</p><p>Setup a <code>port-forward</code> to the Mixer self-monitoring port as described in <a href="#verify-mixer-is-receiving-report-calls">Verify Mixer is receiving Report calls</a>.</p></li><li><p>On the <a href="http://localhost:9093/metrics">Mixer self-monitoring port</a>, search for <code>mixer_config_resolve_count</code>.</p><p>You should find something like:</p><pre><code>mixer_config_resolve_count{error="false",target="details.default.svc.cluster.local"} 56
|
||
mixer_config_resolve_count{error="false",target="ingress.istio-system.svc.cluster.local"} 67
|
||
mixer_config_resolve_count{error="false",target="mongodb.default.svc.cluster.local"} 18
|
||
mixer_config_resolve_count{error="false",target="productpage.default.svc.cluster.local"} 59
|
||
mixer_config_resolve_count{error="false",target="ratings.default.svc.cluster.local"} 26
|
||
mixer_config_resolve_count{error="false",target="reviews.default.svc.cluster.local"} 54
|
||
</code></pre></li><li><p>Validate that there are values for <code>mixer_config_resolve_count</code> where <code>target="<your service>"</code> and <code>error="false"</code>.</p><p>If there are only instances where <code>error="true"</code> where <code>target=<your service></code>, there is likely an issue with Mixer configuration for your service. Logs information is needed to further debug.</p><p>In Kubernetes environments, retrieve the Mixer logs via:</p><pre><code class="language-bash">kubectl -n istio-system logs <mixer pod> mixer
|
||
</code></pre><p>Look for errors related to your configuration or your service in the returned logs.</p></li></ol></li></ol><p>More on viewing Mixer configuration can be found in the <a href="/v0.2/docs/tasks/telemetry/faq.html#how-do-i-see-all-of-the-configuration-for-mixer-">Telemetry FAQ</a>.</p><h3 id="verify-mixer-is-sending-metric-instances-to-the-prometheus-adapter">Verify Mixer is sending metric instances to the Prometheus adapter</h3><ol><li><p>Establish a connection to the Mixer self-monitoring endpoint.</p><p>Setup a <code>port-forward</code> to the Mixer self-monitoring port as described in <a href="#verify-mixer-is-receiving-report-calls">Verify Mixer is receiving Report calls</a>.</p></li><li><p>On the <a href="http://localhost:9093/metrics">Mixer self-monitoring port</a>, search for <code>mixer_adapter_dispatch_count</code>.</p><p>You should find something like:</p><pre><code>mixer_adapter_dispatch_count{adapter="prometheus",error="false",handler="handler.prometheus.istio-system",meshFunction="metric",response_code="OK"} 114
|
||
mixer_adapter_dispatch_count{adapter="prometheus",error="true",handler="handler.prometheus.default",meshFunction="metric",response_code="INTERNAL"} 4
|
||
mixer_adapter_dispatch_count{adapter="stdio",error="false",handler="handler.stdio.istio-system",meshFunction="logentry",response_code="OK"} 104
|
||
</code></pre></li><li><p>Validate that there are values for <code>mixer_adapter_dispatch_count</code> where <code>adapter="prometheus"</code> and <code>error="false"</code>.</p><p>If there are are no recorded dispatches to the Prometheus adapter, there is likely a configuration issue. Please see <a href="#verify-mixer-metrics-configuration-exists">Verify Mixer metrics configuration exists</a>.</p><p>If dispatches to the Prometheus adapter are reporting errors, check the Mixer logs to determine the source of the error. Most likely, there is a configuration issue for the handler listed in <code>mixer_adapter_dispatch_count</code>.</p><p>In Kubernetes environment, check the Mixer logs via:</p><pre><code class="language-bash">kubectl -n istio-system logs <mixer pod> mixer
|
||
</code></pre><p>Filter for lines including something like <code>Report 0 returned with: INTERNAL (1 error occurred:</code> (with some surrounding context) to find more information regarding Report dispatch failures.</p></li></ol><h3 id="verify-prometheus-configuration">Verify Prometheus configuration</h3><ol><li><p>Connect to the Prometheus UI and verify that it can successfully scrape Mixer.</p><p>In Kubernetes environments, setup port-forwarding as follows:</p><pre><code class="language-bash">kubectl -n istio-system port-forward $(kubectl -n istio-system get pod -l app=prometheus -o jsonpath='{.items[0].metadata.name}') 9090:9090 &
|
||
</code></pre></li><li><p>Visit <a href="http://localhost:9090/config">http://localhost:9090/config</a>.</p><p>Confirm that an entry exists that looks like:</p><pre><code class="language-yaml">- job_name: 'istio-mesh'
|
||
# Override the global default and scrape targets from this job every 5 seconds.
|
||
scrape_interval: 5s
|
||
# metrics_path defaults to '/metrics'
|
||
# scheme defaults to 'http'.
|
||
static_configs:
|
||
- targets: ['istio-mixer.istio-system:42422']
|
||
</code></pre></li><li><p>Visit <a href="http://localhost:9090/targets">http://localhost:9090/targets</a>.</p><p>Confirm that target <code>istio-mesh</code> has a status of <strong>UP</strong>.</p></li></ol><h2 id="how-can-i-debug-issues-with-the-service-mesh">How can I debug issues with the service mesh?</h2><h3 id="with-gdb">With <a href="https://www.gnu.org/software/gdb/">GDB</a></h3><p>To debug Istio with <code>gdb</code>, you will need to run the debug images of Envoy / Mixer / Pilot. A recent <code>gdb</code> and the golang extensions (for Mixer/Pilot or other golang components) is required.</p><ol><li><code>kubectl exec -it PODNAME -c [proxy | mixer | pilot]</code></li><li>Find process ID: ps ax</li><li>gdb -p PID binary</li><li>For go: info goroutines, goroutine x bt</li></ol><h3 id="with-tcpdump">With <a href="http://www.tcpdump.org/tcpdump_man.html">Tcpdump</a></h3><p>Tcpdump doesn’t work in the sidecar pod - the container doesn’t run as root. However any other container in the same pod will see all the packets, since the network namespace is shared. <code>iptables</code> will also see the pod-wide config.</p><p>Communication between Envoy and the app happens on 127.0.0.1, and is not encrypted.</p><h2 id="envoy-is-crashing-under-load">Envoy is crashing under load</h2><p>Check your <code>ulimit -a</code>. Many systems have a 1024 open file descriptor limit by default which will cause Envoy to assert and crash with:</p><pre><code class="language-bash">[2017-05-17 03:00:52.735][14236][critical][assert] assert failure: fd_ != -1: external/envoy/source/common/network/connection_impl.cc:58
|
||
</code></pre><p>Make sure to raise your ulimit. Example: <code>ulimit -n 16384</code></p></div></div></div><footer><div class="container"><div class="row"><div class="col-md-2"></div><div class="col-md-3 col-sm-4 col-xs-12 center-block"><ul class="toggle"><p class="header">Docs</p><li><a href="/v0.2/docs/">Welcome</a></li><li><a href="/v0.2/docs/concepts">Concepts</a></li><li><a href="/v0.2/docs/setup">Setup</a></li><li><a href="/v0.2/docs/tasks">Tasks</a></li><li><a href="/v0.2/docs/guides">Guides</a></li><li><a href="/v0.2/docs/reference">Reference</a></li></ul></div><hr class="footer-sections" /><div class="col-md-3 col-sm-4 col-xs-12 center-block"><ul class="toggle"><p class="header">Resources</p><li><a href="/v0.2/faq">Frequently Asked Questions</a></li><li><a href="/v0.2/troubleshooting">Troubleshooting Guide</a></li><li><a href="/v0.2/bugs">Report a Bug</a></li><li><a href="https://github.com/istio/istio.github.io/issues/new?title=Issue with troubleshooting/index.md">Report a Doc Issue</a></li><li><a href="https://github.com/istio/istio.github.io/edit/master/troubleshooting/index.md">Edit This Page on GitHub</a></li></ul></div><hr class="footer-sections" /><div class="col-md-3 col-sm-4 col-xs-12 center-block"><ul class="toggle"><p class="header">Community</p><li><a href="https://groups.google.com/forum/#!forum/istio-users" target="_blank"><span class="group">User</span></a> | <a href="https://groups.google.com/forum/#!forum/istio-dev" target="_blank">Dev</a> | <a href="https://github.com/istio/istio/blob/master/GROUPS.md#working-groups" target="_blank">Working Group Lists</a></li><li><a href="https://twitter.com/IstioMesh" target="_blank"><span class="twitter">Twitter</span></a></li><li><a href="https://github.com/istio/istio" target="_blank"><span class="github">GitHub</span></a></li></ul></div><div class="col-md-1"></div></div><div class="row"><p class="description small text-center"> Istio 0.2, Copyright © 2017 Istio Authors<br> Archived on 12-Nov-2017</p></div></div></footer><script src="https://cdnjs.cloudflare.com/ajax/libs/jquery-validate/1.15.0/jquery.validate.min.js"></script> <script src="https://cdnjs.cloudflare.com/ajax/libs/jquery.form/4.2.1/jquery.form.min.js"></script> <script src="https://cdnjs.cloudflare.com/ajax/libs/jquery-visible/1.2.0/jquery.visible.min.js"></script> <script src="https://cdnjs.cloudflare.com/ajax/libs/jqueryui/1.12.1/jquery-ui.min.js"></script> <script src="https://cdnjs.cloudflare.com/ajax/libs/slick-carousel/1.6.0/slick.min.js"></script> <script src="https://cdnjs.cloudflare.com/ajax/libs/clipboard.js/1.7.1/clipboard.min.js"></script> <script src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/js/bootstrap.min.js"></script> <script src="/v0.2/js/common.js"></script> <script src="/v0.2/js/buttons.js"></script> <script src="/v0.2/js/search.js"></script> <script src="/v0.2/js/prism.js"></script></body></html>
|