---
title: Replicated control planes
description: Install an Istio mesh across multiple Kubernetes clusters with replicated control plane instances.
weight: 2
aliases:
    - /docs/setup/kubernetes/multicluster-install/gateways/
    - /docs/examples/multicluster/gateways/
    - /docs/tasks/multicluster/gateways/
    - /docs/setup/kubernetes/install/multicluster/gateways/
keywords: [kubernetes,multicluster,gateway]
---

Follow this guide to install an Istio
[multicluster deployment](/docs/ops/deployment/deployment-models/#multiple-clusters)
with replicated [control plane](/docs/ops/deployment/deployment-models/#control-plane-models) instances
in every cluster and using gateways to connect services across clusters.

Instead of using a shared Istio control plane to manage the mesh,
in this configuration each cluster has its own Istio control plane
installation, each managing its own endpoints.
All of the clusters are under a shared administrative control for the purposes of
policy enforcement and security.

A single Istio service mesh across the clusters is achieved by replicating
shared services and namespaces and using a common root CA in all of the clusters.
Cross-cluster communication occurs over Istio gateways of the respective clusters.

{{< image width="80%" link="./multicluster-with-gateways.svg" caption="Istio mesh spanning multiple Kubernetes clusters using Istio Gateway to reach remote pods" >}}

## Prerequisites

* Two or more Kubernetes clusters with versions: {{< supported_kubernetes_versions >}}.

* Authority to [deploy the Istio control plane](/docs/setup/install/istioctl/)
  on **each** Kubernetes cluster.

* The IP address of the `istio-ingressgateway` service in each cluster must be accessible
  from every other cluster, ideally using L4 network load balancers (NLB).
  Not all cloud providers support NLBs and some require special annotations to use them,
  so please consult your cloud provider’s documentation for enabling NLBs for
  service object type load balancers. When deploying on platforms without
  NLB support, it may be necessary to modify the health checks for the load
  balancer to register the ingress gateway.

* A **Root CA**. Cross cluster communication requires mutual TLS connection
  between services. To enable mutual TLS communication across clusters, each
  cluster's Citadel will be configured with intermediate CA credentials
  generated by a shared root CA. For illustration purposes, you use a
  sample root CA certificate available in the Istio installation
  under the `samples/certs` directory.

## Deploy the Istio control plane in each cluster

1. Generate intermediate CA certificates for each cluster's Citadel from your
    organization's root CA. The shared root CA enables mutual TLS communication
    across different clusters.

    For illustration purposes, the following instructions use the certificates
    from the Istio samples directory for both clusters. In real world deployments,
    you would likely use a different CA certificate for each cluster, all signed
    by a common root CA.

1. Run the following commands in **every cluster** to deploy an identical Istio control plane
    configuration in all of them.

    {{< tip >}}
    Make sure that the current user has cluster administrator (`cluster-admin`) permissions and grant them if not.
    On the GKE platform, for example, the following command can be used:

    {{< text bash >}}
    $ kubectl create clusterrolebinding cluster-admin-binding --clusterrole=cluster-admin --user="$(gcloud config get-value core/account)"
    {{< /text >}}

    {{< /tip >}}

    * Create a Kubernetes secret for your generated CA certificates using a command similar to the following. See [Certificate Authority (CA) certificates](/docs/tasks/security/citadel-config/plugin-ca-cert/#plugging-in-the-existing-certificate-and-key) for more details.

        {{< warning >}}
        The root and intermediate certificate from the samples directory are widely
        distributed and known.  Do **not** use these certificates in production as
        your clusters would then be open to security vulnerabilities and compromise.
        {{< /warning >}}

        {{< text bash >}}
        $ kubectl create namespace istio-system
        $ kubectl create secret generic cacerts -n istio-system \
            --from-file=@samples/certs/ca-cert.pem@ \
            --from-file=@samples/certs/ca-key.pem@ \
            --from-file=@samples/certs/root-cert.pem@ \
            --from-file=@samples/certs/cert-chain.pem@
        {{< /text >}}

    * Install Istio:

        {{< text bash >}}
        $ istioctl manifest apply \
            -f install/kubernetes/operator/examples/multicluster/values-istio-multicluster-gateways.yaml
        {{< /text >}}

    For further details and customization options, refer to the
    [installation instructions](/docs/setup/install/istioctl/).

## Setup DNS

Providing DNS resolution for services in remote clusters will allow
existing applications to function unmodified, as applications typically
expect to resolve services by their DNS names and access the resulting
IP. Istio itself does not use the DNS for routing requests between
services. Services local to a cluster share a common DNS suffix
(e.g., `svc.cluster.local`). Kubernetes DNS provides DNS resolution for these
services.

To provide a similar setup for services from remote clusters, you name
services from remote clusters in the format
`<name>.<namespace>.global`. Istio also ships with a CoreDNS server that
will provide DNS resolution for these services. In order to utilize this
DNS, Kubernetes' DNS must be configured to `stub a domain` for `.global`.

{{< warning >}}
Some cloud providers have different specific `DNS domain stub` capabilities
and procedures for their Kubernetes services.  Reference the cloud provider's
documentation to determine how to `stub DNS domains` for each unique
environment.  The objective of this bash is to stub a domain for `.global` on
port `53` to reference or proxy the `istiocoredns` service in Istio's service
namespace.
{{< /warning >}}

Create one of the following ConfigMaps, or update an existing one, in each
cluster that will be calling services in remote clusters
(every cluster in the general case):

{{< tabset category-name="platform" >}}
{{< tab name="KubeDNS" category-value="kube-dns" >}}

{{< text bash >}}
$ kubectl apply -f - <<EOF
apiVersion: v1
kind: ConfigMap
metadata:
  name: kube-dns
  namespace: kube-system
data:
  stubDomains: |
    {"global": ["$(kubectl get svc -n istio-system istiocoredns -o jsonpath={.spec.clusterIP})"]}
EOF
{{< /text >}}

{{< /tab >}}

{{< tab name="CoreDNS (< 1.4.0)" category-value="coredns-prev-1.4.0" >}}

{{< text bash >}}
$ kubectl apply -f - <<EOF
apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns
  namespace: kube-system
data:
  Corefile: |
    .:53 {
        errors
        health
        kubernetes cluster.local in-addr.arpa ip6.arpa {
           pods insecure
           upstream
           fallthrough in-addr.arpa ip6.arpa
        }
        prometheus :9153
        proxy . /etc/resolv.conf
        cache 30
        loop
        reload
        loadbalance
    }
    global:53 {
        errors
        cache 30
        proxy . $(kubectl get svc -n istio-system istiocoredns -o jsonpath={.spec.clusterIP})
    }
EOF
{{< /text >}}

{{< /tab >}}

{{< tab name="CoreDNS (>= 1.4.0)" category-value="coredns-after-1.4.0" >}}

{{< text bash >}}
$ kubectl apply -f - <<EOF
apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns
  namespace: kube-system
data:
  Corefile: |
    .:53 {
        errors
        health
        kubernetes cluster.local in-addr.arpa ip6.arpa {
           pods insecure
           upstream
           fallthrough in-addr.arpa ip6.arpa
        }
        prometheus :9153
        forward . /etc/resolv.conf
        cache 30
        loop
        reload
        loadbalance
    }
    global:53 {
        errors
        cache 30
        forward . $(kubectl get svc -n istio-system istiocoredns -o jsonpath={.spec.clusterIP})
    }
EOF
{{< /text >}}

{{< /tab >}}
{{< /tabset >}}

## Configure application services

Every service in a given cluster that needs to be accessed from a different remote
cluster requires a `ServiceEntry` configuration in the remote cluster.
The host used in the service entry should be of the form `<name>.<namespace>.global`
where name and namespace correspond to the service's name and namespace respectively.

To demonstrate cross cluster access, configure the
[sleep service]({{< github_tree >}}/samples/sleep)
running in one cluster to call the [httpbin]({{< github_tree >}}/samples/httpbin) service
running in a second cluster. Before you begin:

* Choose two of your Istio clusters, to be referred to as `cluster1` and `cluster2`.

{{< boilerplate kubectl-multicluster-contexts >}}

### Configure the example services

1. Deploy the `sleep` service in `cluster1`.

    {{< text bash >}}
    $ kubectl create --context=$CTX_CLUSTER1 namespace foo
    $ kubectl label --context=$CTX_CLUSTER1 namespace foo istio-injection=enabled
    $ kubectl apply --context=$CTX_CLUSTER1 -n foo -f @samples/sleep/sleep.yaml@
    $ export SLEEP_POD=$(kubectl get --context=$CTX_CLUSTER1 -n foo pod -l app=sleep -o jsonpath={.items..metadata.name})
    {{< /text >}}

1. Deploy the `httpbin` service in `cluster2`.

    {{< text bash >}}
    $ kubectl create --context=$CTX_CLUSTER2 namespace bar
    $ kubectl label --context=$CTX_CLUSTER2 namespace bar istio-injection=enabled
    $ kubectl apply --context=$CTX_CLUSTER2 -n bar -f @samples/httpbin/httpbin.yaml@
    {{< /text >}}

1. Export the `cluster2` gateway address:

    {{< text bash >}}
    $ export CLUSTER2_GW_ADDR=$(kubectl get --context=$CTX_CLUSTER2 svc --selector=app=istio-ingressgateway \
        -n istio-system -o jsonpath='{.items[0].status.loadBalancer.ingress[0].ip}')
    {{< /text >}}

    This command sets the value to the gateway's public IP, but note that you can set it to
    a DNS name instead, if you have one.

    {{< tip >}}
    If `cluster2` is running in an environment that does not
    support external load balancers, you will need to use a nodePort to access the gateway.
    Instructions for obtaining the IP to use can be found in the
    [Control Ingress Traffic](/docs/tasks/traffic-management/ingress/ingress-control/#determining-the-ingress-ip-and-ports)
    guide. You will also need to change the service entry endpoint port in the following step from 15443
    to its corresponding nodePort
    (i.e., `kubectl --context=$CTX_CLUSTER2 get svc -n istio-system istio-ingressgateway -o=jsonpath='{.spec.ports[?(@.port==15443)].nodePort}'`).
    {{< /tip >}}

1. Create a service entry for the `httpbin` service in `cluster1`.

    To allow `sleep` in `cluster1` to access `httpbin` in `cluster2`, we need to create
    a service entry for it. The host name of the service entry should be of the form
    `<name>.<namespace>.global` where name and namespace correspond to the
    remote service's name and namespace respectively.

    For DNS resolution for services under the `*.global` domain, you need to assign these
    services an IP address.

    {{< tip >}}
    Each service (in the `.global` DNS domain) must have a unique IP within the cluster.
    {{< /tip >}}

    If the global services have actual VIPs, you can use those, but otherwise we suggest
    using IPs from the class E addresses range `240.0.0.0/4`.
    Application traffic for these IPs will be captured by the sidecar and routed to the
    appropriate remote service.

    {{< warning >}}
    Multicast addresses (224.0.0.0 ~ 239.255.255.255) should not be used because there is no route to them by default.
    Loopback addresses (127.0.0.0/8) should also not be used because traffic sent to them may be redirected to the sidecar inbound listener.
    {{< /warning >}}

    {{< text bash >}}
    $ kubectl apply --context=$CTX_CLUSTER1 -n foo -f - <<EOF
    apiVersion: networking.istio.io/v1alpha3
    kind: ServiceEntry
    metadata:
      name: httpbin-bar
    spec:
      hosts:
      # must be of form name.namespace.global
      - httpbin.bar.global
      # Treat remote cluster services as part of the service mesh
      # as all clusters in the service mesh share the same root of trust.
      location: MESH_INTERNAL
      ports:
      - name: http1
        number: 8000
        protocol: http
      resolution: DNS
      addresses:
      # the IP address to which httpbin.bar.global will resolve to
      # must be unique for each remote service, within a given cluster.
      # This address need not be routable. Traffic for this IP will be captured
      # by the sidecar and routed appropriately.
      - 240.0.0.2
      endpoints:
      # This is the routable address of the ingress gateway in cluster2 that
      # sits in front of sleep.foo service. Traffic from the sidecar will be
      # routed to this address.
      - address: ${CLUSTER2_GW_ADDR}
        ports:
          http1: 15443 # Do not change this port value
    EOF
    {{< /text >}}

    The configurations above will result in all traffic in `cluster1` for
    `httpbin.bar.global` on *any port* to be routed to the endpoint
    `<IPofCluster2IngressGateway>:15443` over a mutual TLS connection.

    The gateway for port 15443 is a special SNI-aware Envoy
    preconfigured and installed when you deployed the Istio control plane in the cluster.
    Traffic entering port 15443 will be
    load balanced among pods of the appropriate internal service of the target
    cluster (in this case, `httpbin.bar` in `cluster2`).

    {{< warning >}}
    Do not create a `Gateway` configuration for port 15443.
    {{< /warning >}}

1. Verify that `httpbin` is accessible from the `sleep` service.

    {{< text bash >}}
    $ kubectl exec --context=$CTX_CLUSTER1 $SLEEP_POD -n foo -c sleep -- curl -I httpbin.bar.global:8000/headers
    {{< /text >}}

### Send remote traffic via an egress gateway

If you want to route traffic from `cluster1` via a dedicated egress gateway, instead of directly from the sidecars,
use the following service entry for `httpbin.bar` instead of the one in the previous section.

{{< tip >}}
The egress gateway used in this configuration cannot also be used for other, non inter-cluster, egress traffic.
{{< /tip >}}

If `$CLUSTER2_GW_ADDR` is an IP address, use the `$CLUSTER2_GW_ADDR - IP address` option.  If `$CLUSTER2_GW_ADDR` is a hostname, use the `$CLUSTER2_GW_ADDR - hostname` option.

{{< tabset category-name="profile" >}}

{{< tab name="$CLUSTER2_GW_ADDR - IP address" category-value="option1" >}}
* Export the `cluster1` egress gateway address:

{{< text bash >}}
$ export CLUSTER1_EGW_ADDR=$(kubectl get --context=$CTX_CLUSTER1 svc --selector=app=istio-egressgateway \
    -n istio-system -o yaml -o jsonpath='{.items[0].spec.clusterIP}')
{{< /text >}}

* Apply the httpbin-bar service entry:

{{< text bash >}}
$ kubectl apply --context=$CTX_CLUSTER1 -n foo -f - <<EOF
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: httpbin-bar
spec:
  hosts:
  # must be of form name.namespace.global
  - httpbin.bar.global
  location: MESH_INTERNAL
  ports:
  - name: http1
    number: 8000
    protocol: http
  resolution: STATIC
  addresses:
  - 240.0.0.2
  endpoints:
  - address: ${CLUSTER2_GW_ADDR}
    network: external
    ports:
      http1: 15443 # Do not change this port value
  - address: ${CLUSTER1_EGW_ADDR}
    ports:
      http1: 15443
EOF
{{< /text >}}

{{< /tab >}}

{{< tab name="$CLUSTER2_GW_ADDR - hostname" category-value="option2" >}}
If the `${CLUSTER2_GW_ADDR}` is a hostname, you can use `resolution: DNS` for the endpoint resolution:

{{< text bash >}}
$ kubectl apply --context=$CTX_CLUSTER1 -n foo -f - <<EOF
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: httpbin-bar
spec:
  hosts:
  # must be of form name.namespace.global
  - httpbin.bar.global
  location: MESH_INTERNAL
  ports:
  - name: http1
    number: 8000
    protocol: http
  resolution: DNS
  addresses:
  - 240.0.0.2
  endpoints:
  - address: ${CLUSTER2_GW_ADDR}
    network: external
    ports:
      http1: 15443 # Do not change this port value
  - address: istio-egressgateway.istio-system.svc.cluster.local
    ports:
      http1: 15443
EOF
{{< /text >}}

{{< /tab >}}

{{< /tabset >}}

### Cleanup the example

Execute the following commands to clean up the example services.

* Cleanup `cluster1`:

    {{< text bash >}}
    $ kubectl delete --context=$CTX_CLUSTER1 -n foo -f @samples/sleep/sleep.yaml@
    $ kubectl delete --context=$CTX_CLUSTER1 -n foo serviceentry httpbin-bar
    $ kubectl delete --context=$CTX_CLUSTER1 ns foo
    {{< /text >}}

* Cleanup `cluster2`:

    {{< text bash >}}
    $ kubectl delete --context=$CTX_CLUSTER2 -n bar -f @samples/httpbin/httpbin.yaml@
    $ kubectl delete --context=$CTX_CLUSTER2 ns bar
    {{< /text >}}

* Cleanup `environment variables`:

    {{< text bash >}}
    $ unset SLEEP_POD CLUSTER2_GW_ADDR CLUSTER1_EGW_ADDR CTX_CLUSTER1 CTX_CLUSTER2
    {{< /text >}}

## Version-aware routing to remote services

If the remote service has multiple versions, you can add
labels to the service entry endpoints.
For example:

{{< text bash >}}
$ kubectl apply --context=$CTX_CLUSTER1 -n foo -f - <<EOF
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: httpbin-bar
spec:
  hosts:
  # must be of form name.namespace.global
  - httpbin.bar.global
  location: MESH_INTERNAL
  ports:
  - name: http1
    number: 8000
    protocol: http
  resolution: DNS
  addresses:
  # the IP address to which httpbin.bar.global will resolve to
  # must be unique for each service.
  - 240.0.0.2
  endpoints:
  - address: ${CLUSTER2_GW_ADDR}
    labels:
      cluster: cluster2
    ports:
      http1: 15443 # Do not change this port value
EOF
{{< /text >}}

You can then create virtual services and destination rules
to define subsets of the `httpbin.bar.global` service using the appropriate gateway label selectors.
The instructions are the same as those used for routing to a local service.
See [multicluster version routing](/blog/2019/multicluster-version-routing/)
for a complete example.

## Uninstalling

Uninstall Istio by running the following commands on **every cluster**:

{{< text bash >}}
$ istioctl manifest generate \
    -f install/kubernetes/operator/examples/multicluster/values-istio-multicluster-gateways.yaml \
    | kubectl delete -f -
{{< /text >}}

## Summary

Using Istio gateways, a common root CA, and service entries, you can configure
a single Istio service mesh across multiple Kubernetes clusters.
Once configured this way, traffic can be transparently routed to remote clusters
without any application involvement.
Although this approach requires a certain amount of manual configuration for
remote service access, the service entry creation process could be automated.
