---
title: Multicluster Installation
description: Install an Istio mesh across multiple Kubernetes clusters.
weight: 30
aliases:
- /docs/setup/kubernetes/multicluster-install/
- /docs/setup/kubernetes/multicluster/
- /docs/setup/kubernetes/install/multicluster/
keywords: [kubernetes,multicluster]
test: yes
owner: istio/wg-environments-maintainers
---
Follow this guide to install an Istio {{< gloss >}}service mesh{{< /gloss >}}
that spans multiple {{< gloss "cluster" >}}clusters{{< /gloss >}}.
This guide covers some of the most common concerns when creating a
{{< gloss >}}multicluster{{< /gloss >}} mesh:
- [Network topologies](/docs/ops/deployment/deployment-models#network-models):
one or two networks
- [Control plane topologies](/docs/ops/deployment/deployment-models#control-plane-models):
multiple {{< gloss "primary cluster" >}}primary clusters{{< /gloss >}},
a primary and {{< gloss >}}remote cluster{{< /gloss >}}
Before you begin, review the [deployment models guide](/docs/ops/deployment/deployment-models)
which describes the foundational concepts used throughout this guide.
## Requirements
This guide requires that you have two Kubernetes clusters with any of the
[supported Kubernetes versions](/docs/setup/platform-setup).
The API Server in each cluster must be accessible to the other clusters in the
mesh. Many cloud providers make API Servers publicly accessible via network
load balancers (NLB). If the API Server is not directly accessible, you will
have to modify the installation procedure to enable access. For example, the
[east-west](https://en.wikipedia.org/wiki/East-west_traffic) gateway used in
the multi-network and primary-remote configurations below could also be used
to enable access to the API Server.
## Environment Variables
This guide will refer to two clusters named `cluster1` and `cluster2`. The following
environment variables will be used throughout to simplify the instructions:
Variable | Description
-------- | -----------
`CTX_CLUSTER1` | The context name in the default [Kubernetes configuration file](https://kubernetes.io/docs/tasks/access-application-cluster/configure-access-multiple-clusters/) used for accessing the `cluster1` cluster.
`CTX_CLUSTER2` | The context name in the default [Kubernetes configuration file](https://kubernetes.io/docs/tasks/access-application-cluster/configure-access-multiple-clusters/) used for accessing the `cluster2` cluster.
For example:
{{< text bash >}}
$ export CTX_CLUSTER1=cluster1
$ export CTX_CLUSTER2=cluster2
{{< /text >}}
## Configure Trust
A multicluster service mesh deployment requires that you establish trust
between all clusters in the mesh. Depending on the requirements for your
system, there may be multiple options available for establishing trust.
See [certificate management](/docs/tasks/security/cert-management/) for
detailed descriptions and instructions for all available options.
Depending on which option you choose, the installation instructions for
Istio may change slightly.
This guide will assume that you use a common root to generate intermediate
certificates for each cluster. Follow the [instructions](/docs/tasks/security/cert-management/plugin-ca-cert/)
to generate and push a ca certificate secrets to both the `cluster1` and `cluster2`
clusters.
{{< tip >}}
If you currently have a single cluster with a self-signed CA (as described
in [Getting Started](/docs/setup/getting-started/)), you need to
change the CA using one of the methods described in
[certificate management](/docs/tasks/security/cert-management/). Changing the
CA typically requires reinstalling Istio. The installation instructions
below may have to be altered based on your choice of CA.
{{< /tip >}}
## Install Istio
The steps for installing Istio on multiple clusters depend on your
requirements for network and control plane topology. This section
illustrates the options for two clusters. Meshes that span many clusters may
employ more than one of these options. See
[deployment models](/docs/ops/deployment/deployment-models) for more
information.
{{< tabset category-name="install-istio" >}}
{{< tab name="Multi-Primary" category-value="multi-primary" >}}
The steps that follow will install the Istio control plane on both `cluster1` and
`cluster2`, making each a {{< gloss >}}primary cluster{{< /gloss >}}. Both
clusters reside on the `network1` network, meaning there is direct
connectivity between the pods in both clusters.
Each control plane observes the API Servers in both clusters for endpoints.
Service workloads communicate directly (pod-to-pod) across cluster boundaries.
{{< image width="75%"
link="multi-primary.svg"
caption="Multiple primary clusters on the same network"
>}}
Configure cluster1 as a primary
Create the Istio configuration for `cluster1`:
{{< text bash >}}
$ cat < cluster1.yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
values:
global:
meshID: mesh1
multiCluster:
clusterName: cluster1
network: network1
EOF
{{< /text >}}
Apply the configuration to `cluster1`:
{{< text bash >}}
$ istioctl install --context="${CTX_CLUSTER1}" -f cluster1.yaml
{{< /text >}}
Configure cluster2 as a primary
Create the Istio configuration for `cluster2`:
{{< text bash >}}
$ cat < cluster2.yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
values:
global:
meshID: mesh1
multiCluster:
clusterName: cluster2
network: network1
EOF
{{< /text >}}
Apply the configuration to `cluster2`:
{{< text bash >}}
$ istioctl install --context="${CTX_CLUSTER2}" -f cluster2.yaml
{{< /text >}}
Enable Endpoint Discovery
Install a remote secret in `cluster2` that provides access to `cluster1`’s API server.
{{< text bash >}}
$ istioctl x create-remote-secret \
--context="${CTX_CLUSTER1}" \
--name=cluster1 | \
kubectl apply -f - --context="${CTX_CLUSTER2}"
{{< /text >}}
Install a remote secret in `cluster1` that provides access to `cluster2`’s API server.
{{< text bash >}}
$ istioctl x create-remote-secret \
--context="${CTX_CLUSTER2}" \
--name=cluster2 | \
kubectl apply -f - --context="${CTX_CLUSTER1}"
{{< /text >}}
{{< /tab >}}
{{< tab name="Multi-Primary, Multi-Network" category-value="multi-primary-multi-network" >}}
The following steps will install the Istio control plane on both `cluster1` and
`cluster2`, making each a {{< gloss >}}primary cluster{{< /gloss >}}. Cluster `cluster1` is
on the `network1` network, while `cluster2` is on the `network2` network. This means there
is no direct connectivity between pods across cluster boundaries.
Both `cluster1` and `cluster2` observe the API Servers in each cluster for endpoints.
Service workloads across cluster boundaries communicate indirectly, via
dedicated gateways for [east-west](https://en.wikipedia.org/wiki/East-west_traffic)
traffic. The gateway in each cluster must be reachable from the other cluster.
{{< image width="75%"
link="multi-primary-multi-network.svg"
caption="Multiple primary clusters on separate networks"
>}}
Configure cluster1 as a primary with services exposed
Create the Istio configuration for `cluster1`:
{{< text bash >}}
$ cat < cluster1.yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
values:
global:
meshID: mesh1
multiCluster:
clusterName: cluster1
network: network1
EOF
{{< /text >}}
Apply the configuration to `cluster1`:
{{< text bash >}}
$ istioctl install --context="${CTX_CLUSTER1}" -f cluster1.yaml
{{< /text >}}
Install a gateway in `cluster1` that is dedicated to
[east-west](https://en.wikipedia.org/wiki/East-west_traffic) traffic. By
default, this gateway will be public on the Internet. Production systems may
require additional access restrictions (e.g. via firewall rules) to prevent
external attacks. Check with your cloud vendor to see what options are
available.
{{< text bash >}}
$ MESH=mesh1 CLUSTER=cluster1 NETWORK=network1 \
@samples/multicluster/gen-eastwest-gateway.sh@ | \
istioctl manifest generate -f - | \
kubectl apply --context="${CTX_CLUSTER1}" -f -
{{< /text >}}
Since the clusters are on separate networks, we need to expose all services
(*.local) on the east-west gateway in both clusters. While this gateway is
public on the Internet, services behind it can only be accessed by services
with a trusted mTLS certificate and workload ID, just as if they were on the
same network.
{{< text bash >}}
$ kubectl --context="${CTX_CLUSTER1}" apply -n istio-system -f \
@samples/multicluster/expose-services.yaml@
{{< /text >}}
Configure cluster2 as a primary with services exposed
Create the Istio configuration for `cluster2`:
{{< text bash >}}
$ cat < cluster2.yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
values:
global:
meshID: mesh1
multiCluster:
clusterName: cluster2
network: network2
EOF
{{< /text >}}
Apply the configuration to `cluster2`:
{{< text bash >}}
$ istioctl install --context="${CTX_CLUSTER2}" -f cluster2.yaml
{{< /text >}}
As we did with `cluster1` above, install a gateway in `cluster2` that is dedicated
to east-west traffic and expose user services.
{{< text bash >}}
$ MESH=mesh1 CLUSTER=cluster2 NETWORK=network2 \
@samples/multicluster/gen-eastwest-gateway.sh@ | \
istioctl manifest generate -f - | \
kubectl apply --context="${CTX_CLUSTER2}" -f -
{{< /text >}}
{{< text bash >}}
$ kubectl --context="${CTX_CLUSTER2}" apply -n istio-system -f \
@samples/multicluster/expose-services.yaml@
{{< /text >}}
Enable Endpoint Discovery for cluster1 and cluster2
Install a remote secret in `cluster2` that provides access to `cluster1`’s API server.
{{< text bash >}}
$ istioctl x create-remote-secret \
--context="${CTX_CLUSTER1}" \
--name=cluster1 | \
kubectl apply -f - --context="${CTX_CLUSTER2}"
{{< /text >}}
Install a remote secret in `cluster1` that provides access to `cluster2`’s API server.
{{< text bash >}}
$ istioctl x create-remote-secret \
--context="${CTX_CLUSTER2}" \
--name=cluster2 | \
kubectl apply -f - --context="${CTX_CLUSTER1}"
{{< /text >}}
{{< /tab >}}
{{< tab name="Primary-Remote" category-value="primary-remote" >}}
The following steps will install the Istio control plane on `cluster1` (the
{{< gloss >}}primary cluster{{< /gloss >}}) and configure `cluster2` (the
{{< gloss >}}remote cluster{{< /gloss >}}) to use the control plane in `cluster1`.
Both clusters reside on the `network1` network, meaning there is direct
connectivity between the pods in both clusters.
Cluster `cluster1` will be configured to observe the API Servers in both clusters for
endpoints. In this way, the control plane will be able to provide service
discovery for workloads in both clusters.
Service workloads communicate directly (pod-to-pod) across cluster boundaries.
Services in `cluster2` will reach the control plane in `cluster1` via a
dedicated gateway for [east-west](https://en.wikipedia.org/wiki/East-west_traffic)
traffic.
{{< image width="75%"
link="primary-remote.svg"
caption="Primary and remote clusters on the same network"
>}}
Configure cluster1 as a primary with control plane exposed
Create the Istio configuration for `cluster1`:
{{< text bash >}}
$ cat < cluster1.yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
values:
global:
meshID: mesh1
multiCluster:
clusterName: cluster1
network: network1
EOF
{{< /text >}}
Apply the configuration to `cluster1`:
{{< text bash >}}
$ istioctl install --context="${CTX_CLUSTER1}" -f cluster1.yaml
{{< /text >}}
Install a gateway in `cluster1` that is dedicated to
[east-west](https://en.wikipedia.org/wiki/East-west_traffic) traffic. By
default, this gateway will be public on the Internet. Production systems may
require additional access restrictions (e.g. via firewall rules) to prevent
external attacks. Check with your cloud vendor to see what options are
available.
{{< text bash >}}
$ MESH=mesh1 CLUSTER=cluster1 NETWORK=network1 \
@samples/multicluster/gen-eastwest-gateway.sh@ | \
istioctl manifest generate -f - | \
kubectl apply --context="${CTX_CLUSTER1}" -f -
{{< /text >}}
Before we can install on `cluster2`, we need to first expose the control plane in
`cluster1` so that services in `cluster2` will be able to access service discovery:
{{< text bash >}}
$ kubectl apply --context="${CTX_CLUSTER1}" -f \
@samples/multicluster/expose-istiod.yaml@
{{< /text >}}
Enable API Server Access to cluster2
Before we can configure the remote cluster, we first have to give the control
plane in `cluster1` access to the API Server in `cluster2`. This will do the
following:
- Enables the control plane to authenticate connection requests from
workloads running in `cluster2`. Without API Server access, the control
plane will reject the requests.
- Enables discovery of service endpoints running in `cluster2`.
To provide API Server access to `cluster2`, we generate a remote secret and
apply it to `cluster1`:
{{< text bash >}}
$ istioctl x create-remote-secret \
--context="${CTX_CLUSTER2}" \
--name=cluster2 | \
kubectl apply -f - --context="${CTX_CLUSTER1}"
{{< /text >}}
Configure cluster2 as a remote
Save the address of `cluster1`’s ingress gateway.
{{< text bash >}}
$ export DISCOVERY_ADDRESS=$(kubectl \
--context="${CTX_CLUSTER1}" \
-n istio-system get svc istio-eastwestgateway \
-o jsonpath='{.status.loadBalancer.ingress[0].ip}')
{{< /text >}}
Now create a remote configuration for `cluster2`.
{{< text bash >}}
$ cat < cluster2.yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
values:
global:
meshID: mesh1
multiCluster:
clusterName: cluster2
network: network1
remotePilotAddress: ${DISCOVERY_ADDRESS}
EOF
{{< /text >}}
Apply the configuration to `cluster2`:
{{< text bash >}}
$ istioctl install --context="${CTX_CLUSTER2}" -f cluster2.yaml
{{< /text >}}
{{< /tab >}}
{{< tab name="Primary-Remote, Multi-Network" category-value="primary-remote-multi-network" >}}
The following steps will install the Istio control plane on `cluster1` (the
{{< gloss >}}primary cluster{{< /gloss >}}) and configure `cluster2` (the
{{< gloss >}}remote cluster{{< /gloss >}}) to use the control plane in
`cluster1`. Cluster `cluster1` is on the `network1` network, while `cluster2`
is on the `network2` network. This means there is no direct connectivity
between pods across cluster boundaries.
Cluster `cluster1` will be configured to observe the API Servers in both
clusters for endpoints. In this way, the control plane will be able to
provide service discovery for workloads in both clusters.
Service workloads across cluster boundaries communicate indirectly, via
dedicated gateways for [east-west](https://en.wikipedia.org/wiki/East-west_traffic)
traffic. The gateway in each cluster must be reachable from the other cluster.
Services in `cluster2` will reach the control plane in `cluster1` via the
same east-west gateway.
{{< image width="75%"
link="primary-remote-multi-network.svg"
caption="Primary and remote clusters on separate networks"
>}}
Configure cluster1 as a primary with control plane and services exposed
Create the Istio configuration for `cluster1`:
{{< text bash >}}
$ cat < cluster1.yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
values:
global:
meshID: mesh1
multiCluster:
clusterName: cluster1
network: network1
EOF
{{< /text >}}
Apply the configuration to `cluster1`:
{{< text bash >}}
$ istioctl install --context="${CTX_CLUSTER1}" -f cluster1.yaml
{{< /text >}}
Install a gateway in `cluster1` that is dedicated to east-west traffic. By
default, this gateway will be public on the Internet. Production systems may
require additional access restrictions (e.g. via firewall rules) to prevent
external attacks. Check with your cloud vendor to see what options are
available.
{{< text bash >}}
$ MESH=mesh1 CLUSTER=cluster1 NETWORK=network1 \
@samples/multicluster/gen-eastwest-gateway.sh@ | \
istioctl manifest generate -f - | \
kubectl apply --context="${CTX_CLUSTER1}" -f -
{{< /text >}}
Before we can install on `cluster2`, we need to first expose the control plane in
`cluster1` so that services in `cluster2` will be able to access service discovery:
{{< text bash >}}
$ kubectl apply --context="${CTX_CLUSTER1}" -f \
@samples/multicluster/expose-istiod.yaml@
{{< /text >}}
Since the clusters are on separate networks, we also need to expose all user
services (*.local) on the east-west gateway in both clusters. While this
gateway is public on the Internet, services behind it can only be accessed by
services with a trusted mTLS certificate and workload ID, just as if they were
on the same network.
{{< text bash >}}
$ kubectl --context="${CTX_CLUSTER1}" apply -n istio-system -f \
@samples/multicluster/expose-services.yaml@
{{< /text >}}
Enable API Server Access to cluster2
Before we can configure the remote cluster, we first have to give the control
plane in `cluster1` access to the API Server in `cluster2`. This will do the
following:
- Enables the control plane to authenticate connection requests from
workloads running in `cluster2`. Without API Server access, the control
plane will reject the requests.
- Enables discovery of service endpoints running in `cluster2`.
To provide API Server access to `cluster2`, we generate a remote secret and
apply it to `cluster1`:
{{< text bash >}}
$ istioctl x create-remote-secret \
--context="${CTX_CLUSTER2}" \
--name=cluster2 | \
kubectl apply -f - --context="${CTX_CLUSTER1}"
{{< /text >}}
Configure cluster2 as a remote with services exposed
Save the address of `cluster1`’s ingress gateway.
{{< text bash >}}
$ export DISCOVERY_ADDRESS=$(kubectl \
--context="${CTX_CLUSTER1}" \
-n istio-system get svc istio-eastwestgateway \
-o jsonpath='{.status.loadBalancer.ingress[0].ip}')
{{< /text >}}
Now create a remote configuration on `cluster2`.
{{< text bash >}}
$ cat < cluster2.yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
values:
global:
meshID: mesh1
multiCluster:
clusterName: cluster2
network: network2
remotePilotAddress: ${DISCOVERY_ADDRESS}
EOF
{{< /text >}}
Apply the configuration to `cluster2`:
{{< text bash >}}
$ istioctl install --context="${CTX_CLUSTER2}" -f cluster2.yaml
{{< /text >}}
As we did with `cluster1` above, install a gateway in `cluster2` that is dedicated
to east-west traffic and expose user services.
{{< text bash >}}
$ MESH=mesh1 CLUSTER=cluster2 NETWORK=network2 \
@samples/multicluster/gen-eastwest-gateway.sh@ | \
istioctl manifest generate -f - | \
kubectl apply --context="${CTX_CLUSTER2}" -f -
{{< /text >}}
{{< text bash >}}
$ kubectl --context="${CTX_CLUSTER2}" apply -n istio-system -f \
@samples/multicluster/expose-services.yaml@
{{< /text >}}
{{< /tab >}}
{{< /tabset >}}
## Verify the Installation
To verify that your Istio installation is working as intended, we will deploy
the `HelloWorld` application `V1` to `cluster1` and `V2` to `cluster2`. Upon receiving a
request, `HelloWorld` will include its version in its response.
We will also deploy the `Sleep` container to both clusters. We will use these
pods as the source of requests to the `HelloWorld` service,
simulating in-mesh traffic. Finally, after generating traffic, we will observe
which cluster received the requests.
### Deploy the `HelloWorld` Service
In order to make the `HelloWorld` service callable from any cluster, the DNS
lookup must succeed in each cluster (see
[deployment models](/docs/ops/deployment/deployment-models#dns-with-multiple-clusters)
for details). We will address this by deploying the `HelloWorld` Service to
each cluster in the mesh.
To begin, create the `sample` namespace in each cluster:
{{< text bash >}}
$ kubectl create --context="${CTX_CLUSTER1}" namespace sample
$ kubectl create --context="${CTX_CLUSTER2}" namespace sample
{{< /text >}}
Enable automatic sidecar injection for the `sample` namespace:
{{< text bash >}}
$ kubectl label --context="${CTX_CLUSTER1}" namespace sample \
istio-injection=enabled
$ kubectl label --context="${CTX_CLUSTER2}" namespace sample \
istio-injection=enabled
{{< /text >}}
Create the `HelloWorld` service in both clusters:
{{< text bash >}}
$ kubectl apply --context="${CTX_CLUSTER1}" \
-f @samples/helloworld/helloworld.yaml@ \
-l service=helloworld -n sample
$ kubectl apply --context="${CTX_CLUSTER2}" \
-f @samples/helloworld/helloworld.yaml@ \
-l service=helloworld -n sample
{{< /text >}}
### Deploy `HelloWorld` `V1`
Deploy the `helloworld-v1` application to `cluster1`:
{{< text bash >}}
$ kubectl apply --context="${CTX_CLUSTER1}" \
-f @samples/helloworld/helloworld.yaml@ \
-l version=v1 -n sample
{{< /text >}}
Confirm the `helloworld-v1` pod status:
{{< text bash >}}
$ kubectl get pod --context="${CTX_CLUSTER1}" -n sample -l app=helloworld
NAME READY STATUS RESTARTS AGE
helloworld-v1-86f77cd7bd-cpxhv 2/2 Running 0 40s
{{< /text >}}
Wait until the status of `helloworld-v1` is `Running`.
### Deploy `HelloWorld` `V2`
Deploy the `helloworld-v2` application to `cluster2`:
{{< text bash >}}
$ kubectl apply --context="${CTX_CLUSTER2}" \
-f @samples/helloworld/helloworld.yaml@ \
-l version=v2 -n sample
{{< /text >}}
Confirm the status the `helloworld-v2` pod status:
{{< text bash >}}
$ kubectl get pod --context="${CTX_CLUSTER2}" -n sample -l app=helloworld
NAME READY STATUS RESTARTS AGE
helloworld-v2-758dd55874-6x4t8 2/2 Running 0 40s
{{< /text >}}
Wait until the status of `helloworld-v2` is `Running`.
### Deploy `Sleep`
Deploy the `Sleep` application to both clusters:
{{< text bash >}}
$ kubectl apply --context="${CTX_CLUSTER1}" \
-f @samples/sleep/sleep.yaml@ -n sample
$ kubectl apply --context="${CTX_CLUSTER2}" \
-f @samples/sleep/sleep.yaml@ -n sample
{{< /text >}}
Confirm the status `Sleep` pod on `cluster1`:
{{< text bash >}}
$ kubectl get pod --context="${CTX_CLUSTER1}" -n sample -l app=sleep
NAME READY STATUS RESTARTS AGE
sleep-754684654f-n6bzf 2/2 Running 0 5s
{{< /text >}}
Wait until the status of the `Sleep` pod is `Running`.
Confirm the status of the `Sleep` pod on `cluster2`:
{{< text bash >}}
$ kubectl get pod --context="${CTX_CLUSTER2}" -n sample -l app=sleep
NAME READY STATUS RESTARTS AGE
sleep-754684654f-dzl9j 2/2 Running 0 5s
{{< /text >}}
Wait until the status of the `Sleep` pod is `Running`.
### Verifying Cross-Cluster Traffic
To verify that cross-cluster load balancing works as expected, call the
`HelloWorld` service several times using the `Sleep` pod. To ensure load
balancing is working properly, call the `HelloWorld` service from all
clusters in your deployment.
Send one request from the `Sleep` pod on `cluster1` to the `HelloWorld` service:
{{< text bash >}}
$ kubectl exec --context="${CTX_CLUSTER1}" -n sample -c sleep \
"$(kubectl get pod --context="${CTX_CLUSTER1}" -n sample -l \
app=sleep -o jsonpath='{.items[0].metadata.name}')" \
-- curl helloworld.sample:5000/hello
{{< /text >}}
Repeat this request several times and verify that the `HelloWorld` version
should toggle between `v1` and `v2`:
{{< text plain >}}
Hello version: v2, instance: helloworld-v2-758dd55874-6x4t8
Hello version: v1, instance: helloworld-v1-86f77cd7bd-cpxhv
...
{{< /text >}}
Now repeat this process from the `Sleep` pod on `cluster2`:
{{< text bash >}}
$ kubectl exec --context="${CTX_CLUSTER2}" -n sample -c sleep \
"$(kubectl get pod --context="${CTX_CLUSTER2}" -n sample -l \
app=sleep -o jsonpath='{.items[0].metadata.name}')" \
-- curl helloworld.sample:5000/hello
{{< /text >}}
Repeat this request several times and verify that the `HelloWorld` version
should toggle between `v1` and `v2`:
{{< text plain >}}
Hello version: v2, instance: helloworld-v2-758dd55874-6x4t8
Hello version: v1, instance: helloworld-v1-86f77cd7bd-cpxhv
...
{{< /text >}}
**Congratulations!** You successfully installed and verified Istio on multiple
clusters!