mirror of https://github.com/istio/istio.io.git
64 lines
13 KiB
HTML
64 lines
13 KiB
HTML
<!DOCTYPE html><html lang="en" itemscope itemtype="https://schema.org/WebPage" style="overflow-y: scroll;"><head><meta charset="utf-8"><meta http-equiv="X-UA-Compatible" content="IE=edge"><meta name="viewport" content="width=device-width, initial-scale=1"><meta name="title" content="Troubleshooting Guide"><meta name="og:title" content="Troubleshooting Guide"><meta name="og:image" content="/v0.1/img/logo.png"/><meta name="description" content="Practical advice on practical problems with Istio"><meta name="og:description" content="Practical advice on practical problems with Istio"><title>Istioldie 0.1 / Troubleshooting Guide</title><script> window.ga=window.ga||function(){(ga.q=ga.q||[]).push(arguments)};ga.l=+new Date; ga('create', 'UA-98480406-2', 'auto'); ga('send', 'pageview'); </script> <script async src='https://www.google-analytics.com/analytics.js'></script><link href='https://fonts.googleapis.com/css?family=Roboto:400,100,100italic,300,300italic,400italic,500,500italic,700,700italic,900,900italic' rel='stylesheet' type='text/css'><link rel="alternate" type="application/rss+xml" title="Istio Blog RSS" href="/v0.1/feed.xml"><link rel="apple-touch-icon" href="/v0.1/favicons/apple-touch-icon.png" sizes="180x180"><link rel="icon" type="image/png" href="/v0.1/favicons/android-chrome-96x96.png" sizes="96x96" ><link rel="icon" type="image/png" href="/v0.1/favicons/favicon-32x32.png" sizes="32x32"><link rel="icon" type="image/png" href="/v0.1/favicons/favicon-16x16.png" sizes="16x16"><link rel="manifest" href="/v0.1/favicons/manifest.json"><link rel="mask-icon" href="/v0.1/favicons/safari-pinned-tab.svg" color="#2DA6B0"><meta name="msapplication-TileColor" content="#ffffff"><meta name="msapplication-TileImage" content="/v0.1/favicons/mstile-150x150.png"><link href="//maxcdn.bootstrapcdn.com/bootstrap/3.3.1/css/bootstrap.min.css" rel="stylesheet"><link rel="stylesheet" href="/v0.1/css/all.css"><link rel="stylesheet" href="/v0.1/css/prism.css"><link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/font-awesome/4.4.0/css/font-awesome.min.css"> <script src="https://ajax.googleapis.com/ajax/libs/jquery/1.11.2/jquery.min.js"></script></head><body class="language-unknown"><div class="nav-hero-container" style="z-index: 200000;"><nav id="header-nav" class="navbar navbar-inverse" role="navigation"><div class="container"><div class="row"><div class="col-md-11 nofloat center-block "><div class="navbar-header"> <button type="button" class="hamburger navbar-toggle collapsed" data-toggle="collapse" data-target="#navbar-collapse-1" aria-expanded="false"> <span class="sr-only">Toggle navigation</span> <span class="icon-bar"></span> <span class="icon-bar"></span> <span class="icon-bar"></span> </button> <a class="navbar-brand" href="/v0.1/"><div> <img src="/v0.1/img/logo.png" alt="Istio" width="36px" height="54px"/> <span class="brand-name">Istioldie 0.1</span></div></a></div><div class="collapse navbar-collapse" id="navbar-collapse-1"><ul class="nav navbar-nav navbar-right"><li><a href="/v0.1/about/" >About</a></li><li><a href="/v0.1/docs/" >Docs</a></li><li><a href="/v0.1/blog/" >Blog</a></li><li><a href="/v0.1/community/" >Community</a></li><li><a href="/v0.1/faq/" >FAQ</a></li><li class="dropdown"> <a class="dropdown-toggle" data-toggle="dropdown" href=""> <i class='fa fa-lg fa-cog'></i> <span class="caret"></span> </a><ul class="dropdown-menu"><h6 class="dropdown-header">Other versions of this site</h6><li> <a href="https://istio.io">Current Release</a></li><li> <a href="https://preliminary.istio.io">Next Release</a></li><li> <a href="https://archive.istio.io">Older Releases</a></li></ul></li><li><form name="cse" id="searchbox_demo" class="navbar-form navbar-right" role="search"> <input type="hidden" name="cx" value="013699703217164175118:iwwf17ikgf4" /> <input type="hidden" name="ie" value="utf-8" /> <input type="hidden" name="hl" value="en" /><div class="form-group"><div class="input-group"> <input name="q" class="form-control" type="text" size="30" /><div class="input-group-addon"> <span class="btn-search glyphicon glyphicon-search"></span></div></div></div></form> <script type="text/javascript" src="https://www.google.com/cse/brand?form=searchbox_demo"></script></li></ul></div></div></div></div></nav></div><div class="container troubleshooting"><div class="row"><div class="col-sm-12 col-md-10 col-lg-9 nofloat center-block markdown"><div id="toc" class="toc"></div><h1 id="troubleshooting-guide">Troubleshooting Guide</h1><p>Oh no! You have some problems? Let us help.</p><h2 id="no-traces-appearing-in-zipkin-when-running-istio-locally-on-mac">No traces appearing in Zipkin when running Istio locally on Mac</h2><p>Istio is installed and everything seems to be working except there are no traces showing up in Zipkin when there should be.</p><p>This may be caused by a known <a href="https://github.com/docker/for-mac/issues/1260">Docker issue</a> where the time inside containers may skew significantly from the time on the host machine. If this is the case, when you select a very long date range in Zipin you will see the traces appearing as much as several days too early.</p><p>You can also confirm this problem by comparing the date inside a docker container to outside:</p><pre><code class="language-bash">$ docker run --entrypoint date gcr.io/istio-testing/ubuntu-16-04-slave:latest
|
||
Sun Jun 11 11:44:18 UTC 2017
|
||
$ date -u
|
||
Thu Jun 15 02:25:42 UTC 2017
|
||
</code></pre><p>To fix the problem, you’ll need to shutdown and then restart Docker before reinstalling Istio.</p><h2 id="envoy-wont-connect-to-my-http10-service">Envoy won’t connect to my HTTP/1.0 service</h2><p>Envoy requires HTTP/1.1 or HTTP/2 traffic for upstream services. For example, when using <a href="https://www.nginx.com/">NGINX</a> for serving traffic behind Envoy, you will need to set the <a href="http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_http_version">proxy_http_version</a> directive in your NGINX config to be “1.1”, since the NGINX default is 1.0</p><p>Example config:</p><pre><code>upstream http_backend {
|
||
server 127.0.0.1:8080;
|
||
|
||
keepalive 16;
|
||
}
|
||
|
||
server {
|
||
...
|
||
|
||
location /http/ {
|
||
proxy_pass http://http_backend;
|
||
proxy_http_version 1.1;
|
||
proxy_set_header Connection "";
|
||
...
|
||
}
|
||
}
|
||
</code></pre><h2 id="running-multiple-replicas-of-mixer-does-not-work">Running multiple replicas of Mixer does not work</h2><p>For the 0.1 release, Mixer <strong>must</strong> be configured to run as a single instance within a cluster. We are working on improvements at the protocol, configuration, and deployment levels to support multiple instance and high-availability deployments. We expect to remove this limitation shortly after the initial release.</p><p>Running multiple replicas of Mixer will lead to issues with configuration updates not propagating properly and improperly-enforced quotas (for the memQuota adapter).</p><h2 id="mixers-pod-was-restarted-and-i-lost-my-config-updates">Mixer’s pod was restarted and I lost my config updates</h2><p>For the 0.1 release, Mixer configuration is stored in a local file system-based store. By default, Mixer is not configured to use a Kubernetes persistent volume.</p><p>There are a few possible workarounds:</p><ol><li>Save configuration updates to a common location and script the application of these updates via istioctl.</li><li>Configure a persistent volume and update the Mixer deployment specs to use that volume for the file system store.</li></ol><p>Work is ongoing to provide a highly-available, persistent configuration store for Mixer. We expect this to work to land immediately following the initial release.</p><h3 id="configuring-a-persistent-volume-for-mixer-config">Configuring a persistent volume for Mixer config</h3><p>A Mixer deployment can be modified to use Kubernetes <a href="https://kubernetes.io/docs/tasks/configure-pod-container/configmap/"><code>ConfigMaps</code></a> to provide persistent access to Mixer config. Kubernetes <code>ConfigMaps</code> are preserved across restarts and allow running multiple replicas of Mixer.</p><p>Note: <code>istioctl</code> is not usable for runtime updates to Mixer configuration when this approach is used (the updates will not persist).</p><p>Here is an example ConfigMap:</p><pre><code class="language-yaml">apiVersion: v1
|
||
kind: ConfigMap
|
||
metadata:
|
||
name: mixer-config2
|
||
data:
|
||
adapters.yml: |-
|
||
# Config-map
|
||
subject: global
|
||
adapters:
|
||
…
|
||
descriptors.yml: |-
|
||
subject: namespace:ns
|
||
revision: "2022"
|
||
…
|
||
|
||
apiVersion: v1
|
||
kind: ConfigMap
|
||
metadata:
|
||
name: mixer-configsubjects
|
||
data:
|
||
rules.yml: |-
|
||
…
|
||
…
|
||
|
||
lifecycle:
|
||
postStart:
|
||
exec:
|
||
command: ["/bin/sh", "-c", "cp /etc/opt/mixer2/configroot/scopes/global/adapters.yml /etc/opt/mixer/configroot/scopes/global/adapters.yml; cp /etc/opt/mixer2/configroot/scopes/global/descriptors.yml /etc/opt/mixer/configroot/scopes/global/descriptors.yml; cp /etc/opt/mixer2/configroot/scopes/subjects/rules.yml /etc/opt/mixer/configroot/scopes/global/subjects/global/rules.yml" ]
|
||
|
||
volumeMounts:
|
||
- mountPath: /etc/opt/mixer2/configroot/scopes/subjects
|
||
name: configsubjects
|
||
- mountPath: /etc/opt/mixer2/configroot/scopes/global
|
||
name: config
|
||
volumes:
|
||
- name: config
|
||
configMap:
|
||
name: mixer-config2
|
||
- name: configsubjects
|
||
configMap:
|
||
name: mixer-configsubjects
|
||
</code></pre><h2 id="how-can-i-debug-issues-with-the-service-mesh">How can I debug issues with the service mesh?</h2><h3 id="with-gdb">With <a href="https://www.gnu.org/software/gdb/">GDB</a></h3><p>To debug Istio with <code>gdb</code>, you will need to run the debug images of Envoy / Mixer / Pilot. A recent <code>gdb</code> and the golang extensions (for Mixer/Pilot or other golang components) is required.</p><ol><li><code>kubectl exec -it PODNAME -c [proxy | mixer | pilot]</code></li><li>Find process ID: ps ax</li><li>gdb -p PID binary</li><li>For go: info goroutines, goroutine x bt</li></ol><h3 id="with-tcpdump">With <a href="http://www.tcpdump.org/tcpdump_man.html">Tcpdump</a></h3><p>Tcpdump doesn’t work in the sidecar pod - the container doesn’t run as root. However any other container in the same pod will see all the packets, since the network namespace is shared. <code>iptables</code> will also see the pod-wide config.</p><p>Communication between Envoy and the app happens on 127.0.0.1, and is not encrypted.</p><h2 id="envoy-is-crashing-under-load">Envoy is crashing under load</h2><p>Check your <code>ulimit -a</code>. Many systems have a 1024 open file descriptor limit by default which will cause Envoy to assert and crash with:</p><pre><code class="language-bash">[2017-05-17 03:00:52.735][14236][critical][assert] assert failure: fd_ != -1: external/envoy/source/common/network/connection_impl.cc:58
|
||
</code></pre><p>Make sure to raise your ulimit. Example: <code>ulimit -n 16384</code></p></div></div></div><footer><div class="container"><div class="row"><div class="col-md-2"></div><div class="col-md-3 col-sm-4 col-xs-12 center-block"><ul class="toggle"><p class="header">Docs</p><li><a href="/v0.1/docs/">Welcome</a></li><li><a href="/v0.1/docs/concepts">Concepts</a></li><li><a href="/v0.1/docs/tasks">Tasks</a></li><li><a href="/v0.1/docs/samples">Samples</a></li><li><a href="/v0.1/docs/reference">Reference</a></li></ul></div><hr class="footer-sections" /><div class="col-md-3 col-sm-4 col-xs-12 center-block"><ul class="toggle"><p class="header">Resources</p><li><a href="/v0.1/faq">Frequently Asked Questions</a></li><li><a href="/v0.1/troubleshooting">Troubleshooting Guide</a></li><li><a href="/v0.1/bugs">Report a Bug</a></li><li><a href="https://github.com/istio/istio.github.io/issues/new?title=Issue with troubleshooting/index.md">Report a Doc Issue</a></li><li><a href="https://github.com/istio/istio.github.io/edit/master/troubleshooting/index.md">Edit This Page on GitHub</a></li></ul></div><hr class="footer-sections" /><div class="col-md-3 col-sm-4 col-xs-12 center-block"><ul class="toggle"><p class="header">Community</p><li><a href="https://groups.google.com/forum/#!forum/istio-users" target="_blank"><span class="group">User</span></a> | <a href="https://groups.google.com/forum/#!forum/istio-dev" target="_blank">Dev Mailing Lists</a></li><li><a href="https://twitter.com/IstioMesh" target="_blank"><span class="twitter">Twitter</span></a></li><li><a href="https://github.com/istio/istio" target="_blank"><span class="github">GitHub</span></a></li></ul></div><div class="col-md-1"></div></div><div class="row"><p class="description small text-center"> Copyright © 2017 Istio Authors<br> Istio 0.1<br> Archived on 20-Jul-2017</p></div></div></footer><script src="https://cdnjs.cloudflare.com/ajax/libs/jquery-validate/1.15.0/jquery.validate.min.js"></script> <script src="/v0.1/js/jquery.form.min.js"></script> <script src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.1/js/bootstrap.min.js"></script> <script src="/v0.1/js/slick.min.js"></script> <script src="/v0.1/js/jquery.visible.min.js"></script> <script src="/v0.1/js/common.js" type="text/javascript" charset="utf-8"></script> <script src="/v0.1/js/buttons.js"></script> <script src="/v0.1/js/search.js"></script> <script src="/v0.1/js/prism.js"></script></body></html>
|