mirror of https://github.com/istio/istio.io.git
283 lines
10 KiB
Markdown
283 lines
10 KiB
Markdown
---
|
|
title: Mesh Expansion
|
|
description: Instructions for integrating VMs and bare metal hosts into an Istio mesh deployed on Kubernetes.
|
|
weight: 60
|
|
keywords: [kubernetes,vms]
|
|
---
|
|
|
|
Instructions for integrating VMs and bare metal hosts into an Istio mesh
|
|
deployed on Kubernetes.
|
|
|
|
## Prerequisites
|
|
|
|
* Setup Istio on Kubernetes by following the instructions in the [Installation guide](/docs/setup/kubernetes/quick-start/).
|
|
|
|
* The machine must have IP connectivity to the endpoints in the mesh. This
|
|
typically requires a VPC or a VPN, as well as a container network that
|
|
provides direct (without NAT or firewall deny) routing to the endpoints. The machine
|
|
is not required to have access to the cluster IP addresses assigned by Kubernetes.
|
|
|
|
* The Istio control plane services (Pilot, Mixer, Citadel) and Kubernetes DNS server must be accessible
|
|
from the VMs. This is typically done using an [Internal Load
|
|
Balancer](https://kubernetes.io/docs/concepts/services-networking/service/#internal-load-balancer).
|
|
You can also use NodePort, run Istio components on VMs, or use custom network configurations,
|
|
separate documents will cover these advanced configurations.
|
|
|
|
## Installation steps
|
|
|
|
Setup consists of preparing the mesh for expansion and installing and configuring each VM.
|
|
|
|
An example script to help with Kubernetes setup is available as part of the release bundle and in
|
|
[install/tools/setupMeshEx.sh](https://raw.githubusercontent.com/istio/istio/{{<branch_name>}}/install/tools/setupMeshEx.sh). Check the script content and environment variables supported (like GCP_OPTS).
|
|
|
|
An example script to help configure a machine is available as part of the release bundle and in [install/tools/setupIstioVM.sh](https://raw.githubusercontent.com/istio/istio/{{<branch_name>}}/install/tools/setupIstioVM.sh).
|
|
You should customize it based on your provisioning tools and DNS requirements.
|
|
|
|
### Preparing the Kubernetes cluster for expansion
|
|
|
|
* Setup Internal Load Balancers (ILBs) for Kube DNS, Pilot, Mixer and Citadel. This step is specific to
|
|
each cloud provider, so you may need to edit annotations. You can use an ILB based on Keepalived at
|
|
[here](https://github.com/gyliu513/work/tree/master/k8s/charts/keepalived) for demo or test in case where
|
|
the cloud provider or private cloud (for example IBM Cloud Private) doesn't have load balancer service
|
|
support out of box.
|
|
|
|
```command
|
|
$ kubectl apply -f @install/kubernetes/mesh-expansion.yaml@
|
|
```
|
|
|
|
* Generate the Istio `cluster.env` configuration to be deployed in the VMs. This file contains
|
|
the cluster IP address ranges to intercept.
|
|
|
|
```command
|
|
$ export GCP_OPTS="--zone MY_ZONE --project MY_PROJECT"
|
|
$ @install/tools/setupMeshEx.sh@ generateClusterEnv MY_CLUSTER_NAME
|
|
```
|
|
|
|
Here's an example generated file
|
|
|
|
```command
|
|
$ cat cluster.env
|
|
ISTIO_SERVICE_CIDR=10.63.240.0/20
|
|
```
|
|
|
|
* Generate DNS configuration file to be used in the VMs. This will allow apps on the VM to resolve
|
|
cluster service names, which will be intercepted by the sidecar and forwarded.
|
|
|
|
```command
|
|
$ @install/tools/setupMeshEx.sh@ generateDnsmasq
|
|
```
|
|
|
|
Here's an example generated file
|
|
|
|
```command
|
|
$ cat kubedns
|
|
server=/svc.cluster.local/10.150.0.7
|
|
address=/istio-mixer/10.150.0.8
|
|
address=/istio-pilot/10.150.0.6
|
|
address=/istio-citadel/10.150.0.9
|
|
address=/istio-mixer.istio-system/10.150.0.8
|
|
address=/istio-pilot.istio-system/10.150.0.6
|
|
address=/istio-citadel.istio-system/10.150.0.9
|
|
```
|
|
|
|
### Setting up the machines
|
|
|
|
As an example, you can use the following "all inclusive" script to copy
|
|
and install the setup:
|
|
|
|
```command
|
|
$ export GCP_OPTS="--zone MY_ZONE --project MY_PROJECT"
|
|
$ export SERVICE_NAMESPACE=vm
|
|
```
|
|
|
|
If you are running on a GCE VM, run
|
|
|
|
```command
|
|
$ @install/tools/setupMeshEx.sh@ gceMachineSetup VM_NAME
|
|
```
|
|
|
|
Otherwise, run
|
|
|
|
```command
|
|
$ @install/tools/setupMeshEx.sh@ machineSetup VM_NAME
|
|
```
|
|
|
|
GCE provides better user experience since node agent can always relies on
|
|
GCE metadata instance document to authenticate to Citadel. For everything
|
|
else, e.g., on-prem or raw VM, we have to bootstrap a key/cert as credential,
|
|
which typically has a limited lifetime. And when the cert expires, you have to
|
|
rerun the above command.
|
|
|
|
Or the equivalent manual steps:
|
|
|
|
------ Manual setup steps begin ------
|
|
|
|
* Copy the configuration files and Istio Debian files to each machine joining the cluster.
|
|
Save the files as `/etc/dnsmasq.d/kubedns` and `/var/lib/istio/envoy/cluster.env`.
|
|
|
|
* Configure and verify DNS settings. This may require installing `dnsmasq` and either
|
|
adding it to `/etc/resolv.conf` directly or via DHCP scripts. To verify, check that the VM can resolve
|
|
names and connect to pilot, for example:
|
|
|
|
On the VM/external host:
|
|
|
|
```command
|
|
$ host istio-pilot.istio-system
|
|
```
|
|
|
|
Example generated message:
|
|
|
|
```plain
|
|
$ istio-pilot.istio-system has address 10.150.0.6
|
|
```
|
|
|
|
Check that you can resolve cluster IPs. The actual address will depend on your deployment.
|
|
|
|
```command
|
|
$ host istio-pilot.istio-system.svc.cluster.local.
|
|
```
|
|
|
|
Example generated message:
|
|
|
|
```plain
|
|
istio-pilot.istio-system.svc.cluster.local has address 10.63.247.248
|
|
```
|
|
|
|
Check istio-ingress similarly:
|
|
|
|
```command
|
|
$ host istio-ingress.istio-system.svc.cluster.local.
|
|
```
|
|
|
|
Example generated message:
|
|
|
|
```plain
|
|
istio-ingress.istio-system.svc.cluster.local has address 10.63.243.30
|
|
```
|
|
|
|
* Verify connectivity by checking whether the VM can connect to Pilot and to an endpoint.
|
|
|
|
```command
|
|
$ curl 'http://istio-pilot.istio-system:8080/v1/registration/istio-pilot.istio-system.svc.cluster.local|http-discovery'
|
|
```
|
|
|
|
```json
|
|
{
|
|
"hosts": [
|
|
{
|
|
"ip_address": "10.60.1.4",
|
|
"port": 8080
|
|
}
|
|
]
|
|
}
|
|
```
|
|
|
|
On the VM, use the address above. It will directly connect to the pod running istio-pilot.
|
|
|
|
```command
|
|
$ curl 'http://10.60.1.4:8080/v1/registration/istio-pilot.istio-system.svc.cluster.local|http-discovery'
|
|
```
|
|
|
|
* Extract the initial Istio authentication secrets and copy them to the machine. The default
|
|
installation of Istio includes Citadel and will generate Istio secrets even if
|
|
the automatic 'mTLS'
|
|
setting is disabled (it creates secret for each service account, and the secret
|
|
is named as `istio.<serviceaccount>`). It is recommended that you perform this
|
|
step to make it easy to enable mTLS in the future and to upgrade to a future version
|
|
that will have mTLS enabled by default.
|
|
|
|
`ACCOUNT` defaults to 'default', or `SERVICE_ACCOUNT` environment variable
|
|
`NAMESPACE` defaults to current namespace, or `SERVICE_NAMESPACE` environment variable
|
|
(this step is done by machineSetup)
|
|
On a Mac either `brew install base64` or `set BASE64_DECODE="/usr/bin/base64 -D"`
|
|
|
|
```command
|
|
$ @install/tools/setupMeshEx.sh@ machineCerts ACCOUNT NAMESPACE
|
|
```
|
|
|
|
The generated files (`key.pem`, `root-cert.pem`, `cert-chain.pem`) must be copied to /etc/certs on each machine, readable by istio-proxy.
|
|
|
|
* Install Istio Debian files and start 'istio' and 'istio-auth-node-agent' services.
|
|
Get the debian packages from [GitHub releases](https://github.com/istio/istio/releases) or:
|
|
|
|
```command
|
|
$ source istio.VERSION # defines version and URLs env var
|
|
$ curl -L ${PILOT_DEBIAN_URL}/istio-sidecar.deb > istio-sidecar.deb
|
|
$ dpkg -i istio-sidecar.deb
|
|
$ systemctl start istio
|
|
$ systemctl start istio-auth-node-agent
|
|
```
|
|
|
|
------ Manual setup steps end ------
|
|
|
|
After setup, the machine should be able to access services running in the Kubernetes cluster
|
|
or other mesh expansion machines.
|
|
|
|
```command
|
|
$ curl productpage.bookinfo.svc.cluster.local:9080
|
|
```
|
|
```plain
|
|
... html content ...
|
|
```
|
|
|
|
Check that the processes are running:
|
|
|
|
```command
|
|
$ ps aux |grep istio
|
|
root 6941 0.0 0.2 75392 16820 ? Ssl 21:32 0:00 /usr/local/istio/bin/node_agent --logtostderr
|
|
root 6955 0.0 0.0 49344 3048 ? Ss 21:32 0:00 su -s /bin/bash -c INSTANCE_IP=10.150.0.5 POD_NAME=demo-vm-1 POD_NAMESPACE=default exec /usr/local/bin/pilot-agent proxy > /var/log/istio/istio.log istio-proxy
|
|
istio-p+ 7016 0.0 0.1 215172 12096 ? Ssl 21:32 0:00 /usr/local/bin/pilot-agent proxy
|
|
istio-p+ 7094 4.0 0.3 69540 24800 ? Sl 21:32 0:37 /usr/local/bin/envoy -c /etc/istio/proxy/envoy-rev1.json --restart-epoch 1 --drain-time-s 2 --parent-shutdown-time-s 3 --service-cluster istio-proxy --service-node sidecar~10.150.0.5~demo-vm-1.default~default.svc.cluster.local
|
|
```
|
|
|
|
Istio auth node agent is healthy:
|
|
|
|
```command
|
|
$ sudo systemctl status istio-auth-node-agent
|
|
● istio-auth-node-agent.service - istio-auth-node-agent: The Istio auth node agent
|
|
Loaded: loaded (/lib/systemd/system/istio-auth-node-agent.service; disabled; vendor preset: enabled)
|
|
Active: active (running) since Fri 2017-10-13 21:32:29 UTC; 9s ago
|
|
Docs: https://istio.io/
|
|
Main PID: 6941 (node_agent)
|
|
Tasks: 5
|
|
Memory: 5.9M
|
|
CPU: 92ms
|
|
CGroup: /system.slice/istio-auth-node-agent.service
|
|
└─6941 /usr/local/istio/bin/node_agent --logtostderr
|
|
|
|
Oct 13 21:32:29 demo-vm-1 systemd[1]: Started istio-auth-node-agent: The Istio auth node agent.
|
|
Oct 13 21:32:29 demo-vm-1 node_agent[6941]: I1013 21:32:29.469314 6941 main.go:66] Starting Node Agent
|
|
Oct 13 21:32:29 demo-vm-1 node_agent[6941]: I1013 21:32:29.469365 6941 nodeagent.go:96] Node Agent starts successfully.
|
|
Oct 13 21:32:29 demo-vm-1 node_agent[6941]: I1013 21:32:29.483324 6941 nodeagent.go:112] Sending CSR (retrial #0) ...
|
|
Oct 13 21:32:29 demo-vm-1 node_agent[6941]: I1013 21:32:29.862575 6941 nodeagent.go:128] CSR is approved successfully. Will renew cert in 29m59.137732603s
|
|
```
|
|
|
|
## Running services on a mesh expansion machine
|
|
|
|
* Configure the sidecar to intercept the port. This is configured in `/var/lib/istio/envoy/sidecar.env`,
|
|
using the ISTIO_INBOUND_PORTS environment variable.
|
|
|
|
Example (on the VM running the service):
|
|
|
|
```command
|
|
$ echo "ISTIO_INBOUND_PORTS=3306,8080" > /var/lib/istio/envoy/sidecar.env
|
|
$ systemctl restart istio
|
|
```
|
|
|
|
* Manually configure a selector-less service and endpoints. The `selector-less` service is used for
|
|
services that are not backed by Kubernetes pods.
|
|
|
|
Example, on a machine with permissions to modify Kubernetes services:
|
|
|
|
```command
|
|
$ istioctl -n onprem register mysql 1.2.3.4 3306
|
|
$ istioctl -n onprem register svc1 1.2.3.4 http:7000
|
|
```
|
|
|
|
After the setup, Kubernetes pods and other mesh expansions should be able to access the
|
|
services running on the machine.
|
|
|
|
## What's next
|
|
|
|
* See the [Bookinfo Mesh Expansion](/docs/guides/integrating-vms/) guide.
|