--- title: VPN connectivity description: Install an Istio mesh across multiple Kubernetes clusters with direct network access to remote pods. weight: 5 keywords: [kubernetes,multicluster,federation,vpn] aliases: - /docs/setup/kubernetes/multicluster-install --- Instructions for installing an Istio mesh across multiple clusters when pods in each cluster have direct network access to pods in other clusters. In this configuration, multiple Kubernetes control planes running a remote configuration connect to a **single** Istio control plane. Once one or more remote Kubernetes clusters are connected to the Istio control plane, Envoy can then communicate with the **single** control plane and form a mesh network across multiple clusters. {{< image width="80%" ratio="36.01%" link="./multicluster-with-vpn.svg" caption="Istio mesh spanning multiple Kubernetes clusters with direct network access to remote pods over VPN" >}} ## Prerequisites * Two or more Kubernetes clusters with **1.9 or newer**. * The ability to deploy the [Istio control plane](/docs/setup/kubernetes/quick-start/) on **one** Kubernetes cluster. * The usage of an RFC1918 network, VPN, or alternative more advanced network techniques to meet the following requirements: * Individual cluster Pod CIDR ranges and service CIDR ranges must be unique across the multicluster environment and may not overlap. * All pod CIDRs in every cluster must be routable to each other. * All Kubernetes control plane API servers must be routable to each other. * Helm **2.7.2 or newer**. The use of Tiller is optional. This guide describes how to install a multicluster Istio topology using the manifests and Helm charts provided within the Istio repository. ## Deploy the local Istio control plane Install the [Istio control plane](/docs/setup/kubernetes/quick-start/#installation-steps) on **one** Kubernetes cluster. ## Install the Istio remote on every remote cluster The istio-remote component must be deployed to each remote Kubernetes cluster. There are two approaches to installing the remote. The remote can be installed and managed entirely by Helm and Tiller, or via Helm and `kubectl`. ### Set environment variables for Pod IPs from Istio control plane needed by remote Please wait for the Istio control plane to finish initializing before proceeding to steps in this section. These operations must be run on the Istio control plane cluster to capture the Istio control-plane service endpoints--e.g. Pilot, Policy, and Statsd Pod IP endpoints. If Helm is used with Tiller on each remote, copy the environment variables to each node before using Helm to connect the remote cluster to the Istio control plane. {{< text bash >}} $ export PILOT_POD_IP=$(kubectl -n istio-system get pod -l istio=pilot -o jsonpath='{.items[0].status.podIP}') $ export POLICY_POD_IP=$(kubectl -n istio-system get pod -l istio-mixer-type=policy -o jsonpath='{.items[0].status.podIP}') $ export STATSD_POD_IP=$(kubectl -n istio-system get pod -l istio=statsd-prom-bridge -o jsonpath='{.items[0].status.podIP}') $ export TELEMETRY_POD_IP=$(kubectl -n istio-system get pod -l istio-mixer-type=telemetry -o jsonpath='{.items[0].status.podIP}') $ export ZIPKIN_POD_IP=$(kubectl -n istio-system get pod -l app=jaeger -o jsonpath='{range .items[*]}{.status.podIP}{end}') {{< /text >}} Proceed to one of the options for connecting the remote cluster to the local cluster: * Via [`kubectl` with Helm](#use-kubectl-with-helm-to-connect-the-remote-cluster-to-the-local) * Via [Helm plus Tiller](#alternatively-use-helm-and-tiller-to-connect-the-remote-cluster-to-the-local) * Using *sidecar Injection.* The default behavior is to enable automatic sidecar injection on the remote clusters. For manual sidecar injection refer to the [manual sidecar example](#remote-cluster-manual-sidecar-injection-example) ### Use `kubectl` with Helm to connect the remote cluster to the local 1. Use the `helm template` command on a remote to specify the Istio control plane service endpoints: {{< text bash >}} $ helm template install/kubernetes/helm/istio-remote --namespace istio-system \ --name istio-remote \ --set global.remotePilotAddress=${PILOT_POD_IP} \ --set global.remotePolicyAddress=${POLICY_POD_IP} \ --set global.remoteTelemetryAddress=${TELEMETRY_POD_IP} \ --set global.proxy.envoyStatsd.enabled=true \ --set global.proxy.envoyStatsd.host=${STATSD_POD_IP} \ --set global.remoteZipkinAddress=${ZIPKIN_POD_IP} > $HOME/istio-remote.yaml {{< /text >}} 1. Create a namespace for remote Istio. {{< text bash >}} $ kubectl create ns istio-system {{< /text >}} 1. Instantiate the remote cluster's connection to the Istio control plane: {{< text bash >}} $ kubectl apply -f $HOME/istio-remote.yaml {{< /text >}} 1. Label all the remote cluster's namespaces requiring auto-sidecar injection. The following example labels the `default` namespace. {{< text bash >}} $ kubectl label namespace default istio-injection=enabled {{< /text >}} Repeat for any additional kubernetes namespaces to setup auto-sidecar injection. ### Alternatively use Helm and Tiller to connect the remote cluster to the local 1. If a service account has not already been installed for Helm, please install one: {{< text bash >}} $ kubectl apply -f install/kubernetes/helm/helm-service-account.yaml {{< /text >}} 1. Initialize Helm: {{< text bash >}} $ helm init --service-account tiller {{< /text >}} 1. Install the Helm chart: {{< text bash >}} $ helm install install/kubernetes/helm/istio-remote --name istio-remote --namespace istio-system --set global.remotePilotAddress=${PILOT_POD_IP} --set global.remotePolicyAddress=${POLICY_POD_IP} --set global.remoteTelemetryAddress=${TELEMETRY_POD_IP} --set global.proxy.envoyStatsd.enabled=true --set global.proxy.envoyStatsd.host=${STATSD_POD_IP} --set global.remoteZipkinAddress=${ZIPKIN_POD_IP} {{< /text >}} ### Helm configuration parameters In order for the remote cluster's sidecars interaction with the Istio control plane, the `pilot`, `policy`, `telemetry`, `statsd` and tracing service endpoints need to be configured in the `istio-remote` Helm chart. The chart enables automatic sidecar injection in the remote cluster by default but it can be disabled via a chart variable. The following table describes the `istio-remote` Helm chart's configuration values. | Helm Variable | Accepted Values | Default | Purpose of Value | | --- | --- | --- | --- | | `global.remotePilotAddress` | A valid IP address or hostname | None | Specifies the Istio control plane's pilot Pod IP address or remote cluster DNS resolvable hostname | | `global.remotePolicyAddress` | A valid IP address or hostname | None | Specifies the Istio control plane's policy Pod IP address or remote cluster DNS resolvable hostname | | `global.remoteTelemetryAddress` | A valid IP address or hostname | None | Specifies the Istio control plane's telemetry Pod IP address or remote cluster DNS resolvable hostname | | `global.proxy.envoyStatsd.enabled` | true, false | false | Specifies whether the Istio control plane has Statsd enabled | | `global.proxy.envoyStatsd.host` | A valid IP address or hostname | None | Specifies the Istio control plane's `statsd-prom-bridge` Pod IP address or remote cluster DNS resolvable hostname. Ignored if `global.proxy.envoyStatsd.enabled=false`. | | `global.remoteZipkinAddress` | A valid IP address or hostname | None | Specifies the Istio control plane's tracing application Pod IP address or remote cluster DNS resolvable hostname--e.g. `zipkin` or `jaeger`. | | `sidecarInjectorWebhook.enabled` | true, false | true | Specifies whether to enable automatic sidecar injection on the remote cluster | | `global.remotePilotCreateSvcEndpoint` | true, false | false | If set, a selector-less service and endpoint for `istio-pilot` are created with the `remotePilotAddress` IP, which ensures the `istio-pilot.` is DNS resolvable in the remote cluster. | ## Generate `kubeconfigs` for remote clusters The Istio control plane requires access to all clusters in the mesh to discover services, endpoints, and pod attributes. The following describes how to generate a `kubeconfig` file for a remote cluster to be used by the Istio control plane. The `istio-remote` Helm chart creates a Kubernetes service account named `istio-multi` in the remote cluster with the minimal RBAC access required. The following procedure generates a `kubeconfig` file for the remote cluster using the credentials of the `istio-multi` service account created by the `istio-remote` Helm chart. The following procedure should be performed on each remote cluster to be added to the service mesh. The procedure requires cluster-admin user access to the remote cluster. 1. Prepare environment variables for building the `kubeconfig` file for `ServiceAccount` `istio-multi`: {{< text bash >}} $ export WORK_DIR=$(pwd) $ CLUSTER_NAME=$(kubectl config view --minify=true -o "jsonpath={.clusters[].name}") $ export KUBECFG_FILE=${WORK_DIR}/${CLUSTER_NAME} $ SERVER=$(kubectl config view --minify=true -o "jsonpath={.clusters[].cluster.server}") $ NAMESPACE=istio-system $ SERVICE_ACCOUNT=istio-multi $ SECRET_NAME=$(kubectl get sa ${SERVICE_ACCOUNT} -n ${NAMESPACE} -o jsonpath='{.secrets[].name}') $ CA_DATA=$(kubectl get secret ${SECRET_NAME} -n ${NAMESPACE} -o "jsonpath={.data['ca\.crt']}") $ TOKEN=$(kubectl get secret ${SECRET_NAME} -n ${NAMESPACE} -o "jsonpath={.data['token']}" | base64 --decode) {{< /text >}} __NOTE__: An alternative to `base64 --decode` is `openssl enc -d -base64 -A` on many systems. 1. Create a `kubeconfig` file in the working directory for the `ServiceAccount` `istio-multi`: {{< text bash >}} $ cat < ${KUBECFG_FILE} apiVersion: v1 clusters: - cluster: certificate-authority-data: ${CA_DATA} server: ${SERVER} name: ${CLUSTER_NAME} contexts: - context: cluster: ${CLUSTER_NAME} user: ${CLUSTER_NAME} name: ${CLUSTER_NAME} current-context: ${CLUSTER_NAME} kind: Config preferences: {} users: - name: ${CLUSTER_NAME} user: token: ${TOKEN} EOF {{< /text >}} 1. _(Optional)_ Create file with environment variables for creating the remote cluster secret: {{< text bash >}} $ cat < remote_cluster_env_vars export CLUSTER_NAME=${CLUSTER_NAME} export KUBECFG_FILE=${KUBECFG_FILE} export NAMESPACE=${NAMESPACE} EOF {{< /text >}} At this point, the remote clusters' `kubeconfig` files have been created in the current directory. The filename for a cluster is the same as the original `kubeconfig` cluster name. ## Instantiate the credentials for each remote cluster Execute this work on the cluster running the Istio control plane using the `WORK_DIR`, `CLUSTER_NAME`, and `NAMESPACE` environment values set during the [Generate kubeconfig for remote clusters](#generate-kubeconfigs-for-remote-clusters) steps. * _(Optional)_ Source the environment variables file created for the remote cluster secret: {{< text bash >}} $ source remote_cluster_env_vars {{< /text >}} Istio can be installed in a different namespace other than istio-system. The local cluster running the Istio control plane does not need it's secrets stored and labeled. The local node is always aware of its Kubernetes credentials, but the local node is not aware of the remote nodes' credentials. Create a secret and label it properly for each remote cluster: {{< text bash >}} $ kubectl create secret generic ${CLUSTER_NAME} --from-file ${KUBECFG_FILE} -n ${NAMESPACE} $ kubectl label secret ${CLUSTER_NAME} istio/multiCluster=true -n ${NAMESPACE} {{< /text >}} {{< warning_icon >}} Kubernetes secret data keys have to conform to `DNS-1123 subdomain` [format](https://tools.ietf.org/html/rfc1123#page-13), so the filename can't have underscores for example. To resolve any issue you can simply change the filename to conform to the format. ## Uninstalling > The uninstall method must match the installation method (`Helm and kubectl` or `Helm and Tiller` based). ### Use `kubectl` to uninstall istio-remote {{< text bash >}} $ kubectl delete -f $HOME/istio-remote.yaml {{< /text >}} ### Alternatively use Helm and Tiller to uninstall istio-remote {{< text bash >}} $ helm delete --purge istio-remote {{< /text >}} ## Remote cluster manual sidecar injection example The following example shows how to use the `helm template` command to generate the manifest for the remote cluster with automatic sidecar injection disabled. Additionally, the example indicates how to use the remote clusters' configmaps with the `istioctl kube-inject` command to generate any application manifests for the remote cluster. The following procedure is to be performed against the remote cluster. > The endpoint IP environment variables need to be set as in the [above section](#set-environment-variables-for-pod-ips-from-istio-control-plane-needed-by-remote) 1. Use the `helm template` command on a remote to specify the Istio control plane service endpoints: {{< text bash >}} $ helm template install/kubernetes/helm/istio-remote --namespace istio-system --name istio-remote --set global.remotePilotAddress=${PILOT_POD_IP} --set global.remotePolicyAddress=${POLICY_POD_IP} --set global.remoteTelemetryAddress=${TELEMETRY_POD_IP} --set global.proxy.envoyStatsd.enabled=true --set global.proxy.envoyStatsd.host=${STATSD_POD_IP} --set global.remoteZipkinAddress=${ZIPKIN_POD_IP} --set sidecarInjectorWebhook.enabled=false > $HOME/istio-remote_noautoinj.yaml {{< /text >}} 1. Create a namespace for remote Istio. {{< text bash >}} $ kubectl create ns istio-system {{< /text >}} 1. Instantiate the remote cluster's connection to the Istio control plane: {{< text bash >}} $ kubectl apply -f $HOME/istio-remote_noautoinj.yaml {{< /text >}} 1. [Generate kubeconfig for remote clusters](#generate-kubeconfigs-for-remote-clusters) 1. [Instantiate the credentials for each remote cluster](#instantiate-the-credentials-for-each-remote-cluster) ### Manually inject sidecars into application manifests The following is an example `istioctl` command to inject sidecars into application manifests. The commands should be run in a shell with `kubeconfig` context setup for the remote cluster. {{< text bash >}} $ ORIGINAL_SVC_MANIFEST=mysvc-v1.yaml $ istioctl kube-inject --injectConfigMapName istio-sidecar-injector --meshConfigMapName istio -f ${ORIGINAL_SVC_MANIFEST} | kubectl apply -f - {{< /text >}} ## Accessing services from different clusters Kubernetes resolves DNS on a cluster basis. Because DNS resolution is tied to the cluster, you must define the service object in every cluster where a client runs, regardless of the location of the service's endpoints. To ensure this is the case, duplicate the service object to every cluster using `kubectl`. Duplication ensures Kubernetes can resolve the service name in any cluster. Since the service objects are defined in a namespace, you must define the namespace if it doesn't exist, and include it in the service definitions in all clusters. ## Deployment considerations The above procedure provides a simple and step by step guide to deploy a multicluster environment. A production environment might require additional steps or more complex deployment options. The procedure gathers the endpoint IPs of Istio services and uses them to invoke Helm. This create Istio services on the remote clusters. As part of creating those services and endpoints in the remote cluster Kubernetes will add DNS entries into kube-dns. This allows kube-dns in the remote clusters to resolve the Istio service names for all envoy sidecars in those remote clusters. Since Kubernetes pods don't have stable IPs, restart of any Istio service pod in the control plane cluster will cause its endpoint to be changed. Therefore, any connection made from remote clusters to that endpoint will be broken. This is documented in [Istio issue #4822](https://github.com/istio/istio/issues/4822) There are a number of ways to either avoid or resolve this scenario. This section provides a high level overview of these options. * Update the DNS entries * Use a load balancer service type * Expose the Istio services via a gateway ### Update the DNS entries Upon any failure or pod restart kube-dns on the remote clusters can be updated with the correct endpoint mappings for the Istio services. There are a number of ways this can be done. The most obvious is to rerun the Helm install in the remote cluster after the Istio services on the control plane cluster have restarted. ### Use load balance service type In Kubernetes, you can declare a service with a service type to be [`LoadBalancer`](https://kubernetes.io/docs/concepts/services-networking/service/#publishing-services-service-types). A simple solution to the pod restart issue is to use load balancers for the Istio services. You can then use the load balancer IPs as the Istio services's endpoint IPs to configure the remote clusters. You may need balancer IPs for these Istio services: `istio-pilot, istio-telemetry, istio-policy, istio-statsd-prom-bridge, zipkin` Currently, Istio installation doesn't provide an option to specify service types for the Istio services. But you can modify the Istio Helm charts or the Istio manifests yourself. ### Expose the Istio services via a gateway This uses the Istio Ingress gateway functionality. The remote clusters have the `istio-pilot, istio-telemetry, istio-policy, istio-statsd-prom-bridge, zipkin` services pointing to the load balanced IP of the Istio ingress. All the services can point to the same IP. The ingress gateway is then provided with destination rules to reach the proper Istio service in the main cluster. Within this option there are 2 sub-options. One is to re-use the default Istio ingress gateway installed with the provided manifests or Helm charts. The other option is to create another Istio ingress gateway specifically for multicluster. ## Security Istio supports deployment of mutual TLS between the control plane components as well as between sidecar injected application pods. ### Control plane security The steps to enable control plane security are as follows: 1. Istio control plane cluster deployed with 1. control plane security enabled 1. `citadel` certificate self signing disabled 1. a secret named `cacerts` in the Istio control plane namespace with the [CA certificates](/docs/tasks/security/plugin-ca-cert/#plugging-in-the-existing-certificate-and-key) 1. Istio remote clusters deployed with 1. control plane security enabled 1. `citadel` certificate self signing disabled 1. a secret named `cacerts` in the Istio control plane namespace with the [CA certificates](/docs/tasks/security/plugin-ca-cert/#plugging-in-the-existing-certificate-and-key) 1. The CA certificate for the remote clusters needs to be signed by the same CA or root CA as the main cluster. 1. Istio pilot service hostname resolvable via DNS 1. Required because Istio configures the sidecar to verify the certificate subject names using the `istio-pilot.` subject name format. 1. Control plane IPs or resolvable host names set ### Mutual TLS between application pods The steps to enable mutual TLS for all application pods are as follows: 1. Istio control plane cluster deployed with 1. Global mutual TLS enabled 1. `citadel` certificate self signing disabled 1. a secret named `cacerts` in the Istio control plane namespace with the [CA certificates](/docs/tasks/security/plugin-ca-cert/#plugging-in-the-existing-certificate-and-key) 1. Istio remote clusters deployed with 1. Global mutual TLS enabled 1. `citadel` certificate self signing disabled 1. a secret named `cacerts` in the Istio control plane namespace with the [CA certificates](/docs/tasks/security/plugin-ca-cert/#plugging-in-the-existing-certificate-and-key) 1. The CA certificate for the remote clusters needs to be signed by the same CA or root CA as the main cluster. > The CA certificate steps are identical for both control plane security and application pod security steps. ### Example deployment The following is an example procedure to install Istio with both control plane mutual TLS and application pod mutual TLS enabled. The example sets up a remote cluster with a selector-less service and endpoint for `istio-pilot` to allow the remote sidecars to resolve the `istio-pilot.istio-system` hostname via its local Kubernetes DNS. 1. *Primary Cluster.* Deployment of the Istio control plane cluster 1. Create the `cacerts` secret from the Istio samples certificate in the `istio-system` namespace: {{< text bash >}} $ kubectl create ns istio-system $ kubectl create secret generic cacerts -n istio-system --from-file=samples/certs/ca-cert.pem --from-file=samples/certs/ca-key.pem --from-file=samples/certs/root-cert.pem --from-file=samples/certs/cert-chain.pem {{< /text >}} 1. Deploy the Istio control plane with control plane and application pod security enabled {{< text bash >}} $ helm template --namespace=istio-system \ --values install/kubernetes/helm/istio/values.yaml \ --set global.mtls.enabled=true \ --set security.selfSigned=false \ --set global.controlPlaneSecurityEnabled=true \ install/kubernetes/helm/istio > ${HOME}/istio-auth.yaml $ kubectl apply -f ${HOME}/istio-auth.yaml {{< /text >}} 1. *Remote Cluster.* Deployment of remote cluster's istio components 1. Create the `cacerts` secret from the Istio samples certificate in the `istio-system` namespace: {{< text bash >}} $ kubectl create ns istio-system $ kubectl create secret generic cacerts -n istio-system --from-file=samples/certs/ca-cert.pem --from-file=samples/certs/ca-key.pem --from-file=samples/certs/root-cert.pem --from-file=samples/certs/cert-chain.pem {{< /text >}} 1. Set endpoint IP environment variables as in the [setting environment variables](#set-environment-variables-for-pod-ips-from-istio-control-plane-needed-by-remote) section 1. Deploy the remote cluster's components with control plane and application pod security enabled. Also, enable creation of the `istio-pilot` selector-less service and endpoint to get a DNS entry in the remote cluster. {{< text bash >}} $ helm template install/kubernetes/helm/istio-remote \ --name istio-remote \ --namespace=istio-system \ --set global.mtls.enabled=true \ --set security.selfSigned=false \ --set global.controlPlaneSecurityEnabled=true \ --set global.remotePilotCreateSvcEndpoint=true \ --set global.remotePilotAddress=${PILOT_POD_IP} \ --set global.remotePolicyAddress=${POLICY_POD_IP} \ --set global.remoteTelemetryAddress=${TELEMETRY_POD_IP} \ --set global.proxy.envoyStatsd.enabled=true \ --set global.proxy.envoyStatsd.host=${STATSD_POD_IP} > ${HOME}/istio-remote-auth.yaml $ kubectl apply -f ${HOME}/istio-remote-auth.yaml {{< /text >}} 1. [Generate kubeconfig for remote cluster](#generate-kubeconfigs-for-remote-clusters) 1. *Primary Cluster.* [Instantiate the credentials for each remote cluster](#instantiate-the-credentials-for-each-remote-cluster) At this point all of the Istio components in both clusters are configured for mutual TLS between application sidecars and the control plane components as well as between the other application sidecars.