mirror of https://github.com/istio/istio.io.git
Add troubleshooting guide for VMs (#8868)
* Add troubleshooting guide for VMs * Update content/en/docs/ops/diagnostic-tools/virtual-machines/index.md Co-authored-by: Jimmy Song <rootsongjc@gmail.com> Co-authored-by: Jimmy Song <rootsongjc@gmail.com>
This commit is contained in:
John Howard
GitHub
parent
eb318dd57a
commit
fec7e43e83
|
|
@ -0,0 +1,121 @@
|
||||||
|
---
|
||||||
|
title: Debugging Virtual Machines
|
||||||
|
description: Describes tools and techniques to diagnose issues with Virtual Machines.
|
||||||
|
weight: 20
|
||||||
|
keywords: [debug,virtual-machines,envoy]
|
||||||
|
owner: istio/wg-environments-maintainers
|
||||||
|
test: n/a
|
||||||
|
---
|
||||||
|
|
||||||
|
This page describes how to troubleshoot issues with Istio deployed to Virtual Machines.
|
||||||
|
Before reading this, you should take the steps in [Virtual Machine Installation](/docs/setup/install/virtual-machine/).
|
||||||
|
|
||||||
|
Troubleshooting an Istio Virtual Machine installation is similar to troubleshooting issues with proxies running inside Kubernetes, but there are some key differences to be aware of.
|
||||||
|
|
||||||
|
While much of the same information is available on both platforms, accessing this information differs.
|
||||||
|
|
||||||
|
## Monitoring health
|
||||||
|
|
||||||
|
The Istio sidecar is typically run as a `systemd` unit. To ensure its running properly, you can check that status:
|
||||||
|
|
||||||
|
{{< text bash >}}
|
||||||
|
$ systemctl status istio
|
||||||
|
{{< /text >}}
|
||||||
|
|
||||||
|
Additionally, the sidecar health can be programmatically check at its health endpoint:
|
||||||
|
|
||||||
|
{{< text bash >}}
|
||||||
|
$ curl localhost:15021/healthz/ready -I
|
||||||
|
{{< /text >}}
|
||||||
|
|
||||||
|
## Logs
|
||||||
|
|
||||||
|
Logs for the Istio proxy can be found in a few places.
|
||||||
|
|
||||||
|
To access the `systemd` logs, which has details about the initialization of the proxy:
|
||||||
|
|
||||||
|
{{< text bash >}}
|
||||||
|
$ journalctl -f -u istio -n 1000
|
||||||
|
{{< /text >}}
|
||||||
|
|
||||||
|
The proxy will redirect `stderr` and `stdout` to `/var/log/istio/istio.err.log` and `/var/log/istio/istio.log`, respectively.
|
||||||
|
To view these in a format similar to `kubectl`:
|
||||||
|
|
||||||
|
{{< text bash >}}
|
||||||
|
$ tail /var/log/istio/istio.err.log /var/log/istio/istio.log -Fq -n 100
|
||||||
|
{{< /text >}}
|
||||||
|
|
||||||
|
Log levels can be modified by changing the `cluster.env` configuration file. Make sure to restart `istio` if it is already running:
|
||||||
|
|
||||||
|
{{< text bash >}}
|
||||||
|
$ echo "ISTIO_AGENT_FLAGS=\"--log_output_level=dns:debug --proxyLogLevel=debug\"" >> /var/lib/istio/envoy/cluster.env
|
||||||
|
$ systemctl restart istio
|
||||||
|
{{< /text >}}
|
||||||
|
|
||||||
|
## Iptables
|
||||||
|
|
||||||
|
To ensure `iptables` rules have been successfully applied:
|
||||||
|
|
||||||
|
{{< text bash >}}
|
||||||
|
$ sudo iptables-save
|
||||||
|
...
|
||||||
|
-A ISTIO_OUTPUT -d 127.0.0.1/32 -j RETURN
|
||||||
|
-A ISTIO_OUTPUT -j ISTIO_REDIRECT
|
||||||
|
{{< /text >}}
|
||||||
|
|
||||||
|
## Istioctl
|
||||||
|
|
||||||
|
Most `istioctl` commands will function properly with virtual machines. For example, `istioctl proxy-status` can be used to view all connected proxies:
|
||||||
|
|
||||||
|
{{< text bash >}}
|
||||||
|
$ istioctl proxy-status
|
||||||
|
NAME CDS LDS EDS RDS ISTIOD VERSION
|
||||||
|
vm-1.default SYNCED SYNCED SYNCED SYNCED istiod-789ffff8-f2fkt {{< istio_full_version >}}
|
||||||
|
{{< /text >}}
|
||||||
|
|
||||||
|
However, `istioctl proxy-config` relies on functionality in Kubernetes to connect to a proxy, which will not work for virtual machines.
|
||||||
|
Instead, a file containing the configuration dump from Envoy can be passed. For example:
|
||||||
|
|
||||||
|
{{< text bash >}}
|
||||||
|
$ curl -s localhost:15000/config_dump | istioctl proxy-config clusters --file -
|
||||||
|
SERVICE FQDN PORT SUBSET DIRECTION TYPE
|
||||||
|
istiod.istio-system.svc.cluster.local 443 - outbound EDS
|
||||||
|
istiod.istio-system.svc.cluster.local 15010 - outbound EDS
|
||||||
|
istiod.istio-system.svc.cluster.local 15012 - outbound EDS
|
||||||
|
istiod.istio-system.svc.cluster.local 15014 - outbound EDS
|
||||||
|
{{< /text >}}
|
||||||
|
|
||||||
|
## Automatic registration
|
||||||
|
|
||||||
|
When a virtual machine connects to Istiod, a `WorkloadEntry` will automatically be created. This enables
|
||||||
|
the virtual machine to become a part of a `Service`, similar to an `Endpoint` in Kubernetes.
|
||||||
|
|
||||||
|
To check these are created correctly:
|
||||||
|
|
||||||
|
{{< text bash >}}
|
||||||
|
$ kubectl get workloadentries
|
||||||
|
NAME AGE ADDRESS
|
||||||
|
vm-10.128.0.50 14m 10.128.0.50
|
||||||
|
{{< /text >}}
|
||||||
|
|
||||||
|
## Certificates
|
||||||
|
|
||||||
|
Virtual machines handle certificates differently than Kubernetes Pods, which use a Kubernetes-provided service account token
|
||||||
|
to authenticate and renew mTLS certificates. Instead, existing mTLS credentials are used to authenticate with the certificate authority and
|
||||||
|
renew certificates.
|
||||||
|
|
||||||
|
The status of these certificates can be viewed in the same way as in Kubernetes:
|
||||||
|
|
||||||
|
{{< text bash >}}
|
||||||
|
$ curl -s localhost:15000/config_dump | ./istioctl proxy-config secret --file -
|
||||||
|
RESOURCE NAME TYPE STATUS VALID CERT SERIAL NUMBER NOT AFTER NOT BEFORE
|
||||||
|
default Cert Chain ACTIVE true 251932493344649542420616421203546836446 2021-01-29T18:07:21Z 2021-01-28T18:07:21Z
|
||||||
|
ROOTCA CA ACTIVE true 81663936513052336343895977765039160718 2031-01-26T17:54:44Z 2021-01-28T17:54:44Z
|
||||||
|
{{< /text >}}
|
||||||
|
|
||||||
|
Additionally, these are persisted to disk to ensure downtime or restarts do not lose state.
|
||||||
|
|
||||||
|
{{< text bash >}}
|
||||||
|
$ ls /etc/certs
|
||||||
|
cert-chain.pem key.pem root-cert.pem
|
||||||
|
{{< /text >}}
|
||||||
Loading…
Reference in New Issue