From 3f0eb787e530ce507ea9a2ef1d78b0f7e1bca887 Mon Sep 17 00:00:00 2001 From: Shriram Rajagopalan Date: Fri, 12 Oct 2018 13:46:10 -0400 Subject: [PATCH] multi-cluster over gateways (#2713) * some diagrams for multi-cluster Signed-off-by: Shriram Rajagopalan * nits Signed-off-by: Shriram Rajagopalan * tweaks Signed-off-by: Shriram Rajagopalan * section index Signed-off-by: Shriram Rajagopalan * remove ... Signed-off-by: Shriram Rajagopalan * page description ends with a period Signed-off-by: Shriram Rajagopalan * completing the doc Signed-off-by: Shriram Rajagopalan * flat network Signed-off-by: Shriram Rajagopalan * multicluster to federation Signed-off-by: Shriram Rajagopalan * federation Signed-off-by: Shriram Rajagopalan * multicluster-install Signed-off-by: Shriram Rajagopalan * fix section title Signed-off-by: Shriram Rajagopalan * proofer Signed-off-by: Shriram Rajagopalan * html proofing again Signed-off-by: Shriram Rajagopalan * final nit Signed-off-by: Shriram Rajagopalan * nits Signed-off-by: Shriram Rajagopalan * spell Signed-off-by: Shriram Rajagopalan * Address review comments and other improvements * a couple more nits * fix build error * add missing end tag * fix lint errors * mention decentralized pilot --- .spelling | 1 + content/docs/examples/multicluster/_index.md | 4 +- .../docs/examples/multicluster/gke/index.md | 5 +- .../docs/examples/multicluster/icp/index.md | 4 +- .../examples/multicluster/iks-icp/index.md | 4 +- .../kubernetes/multicluster-install/_index.md | 7 + .../multicluster-install/gateways/index.md | 278 ++++++++++++++++++ .../gateways/multicluster-with-gateways.svg | 1 + .../multicluster-install/{ => vpn}/index.md | 33 ++- .../vpn/multicluster-with-vpn.svg | 1 + 10 files changed, 316 insertions(+), 22 deletions(-) create mode 100644 content/docs/setup/kubernetes/multicluster-install/_index.md create mode 100644 content/docs/setup/kubernetes/multicluster-install/gateways/index.md create mode 100644 content/docs/setup/kubernetes/multicluster-install/gateways/multicluster-with-gateways.svg rename content/docs/setup/kubernetes/multicluster-install/{ => vpn}/index.md (96%) create mode 100644 content/docs/setup/kubernetes/multicluster-install/vpn/multicluster-with-vpn.svg diff --git a/.spelling b/.spelling index 1476a5fb42..30c5f71e8a 100644 --- a/.spelling +++ b/.spelling @@ -165,6 +165,7 @@ Undeploy VMware VM-based VMs +VPN ValueType WeaveWorks WebSocket diff --git a/content/docs/examples/multicluster/_index.md b/content/docs/examples/multicluster/_index.md index 95afe1ba6c..116b4f631f 100644 --- a/content/docs/examples/multicluster/_index.md +++ b/content/docs/examples/multicluster/_index.md @@ -1,6 +1,6 @@ --- -title: Enabling multiclusters -description: A variety of fully working multicluster examples for Istio that you can experiment with. +title: Configuring Istio mesh across multiple clusters in same network +description: A variety of fully working VPN-based multicluster examples for Istio that you can experiment with. weight: 100 keywords: [multicluster] --- diff --git a/content/docs/examples/multicluster/gke/index.md b/content/docs/examples/multicluster/gke/index.md index bfc2ee2ea7..382701c2ff 100644 --- a/content/docs/examples/multicluster/gke/index.md +++ b/content/docs/examples/multicluster/gke/index.md @@ -7,7 +7,8 @@ keywords: [kubernetes,multicluster] This example demonstrates how to use Istio's multicluster feature to join 2 [Google Kubernetes Engine](https://cloud.google.com/kubernetes-engine/) clusters together, -using the [multicluster installation instructions](/docs/setup/kubernetes/multicluster-install/). +using the +[VPN-based multicluster installation instructions](/docs/setup/kubernetes/multicluster-install/vpn/). ## Before you begin @@ -317,7 +318,7 @@ $ kubectl label secret ${CLUSTER_NAME} istio/multiCluster=true -n ${NAMESPACE} ## Uninstalling The following should be done in addition to the uninstall of Istio as described in the -[Kubernetes multicluster installation instructions](/docs/setup/kubernetes/multicluster-install/): +[VPN-based multicluster uninstall section](/docs/setup/kubernetes/multicluster-install/vpn/): 1. Delete the Google Cloud firewall rule: diff --git a/content/docs/examples/multicluster/icp/index.md b/content/docs/examples/multicluster/icp/index.md index 55a1cee6ab..a5dddda9b0 100644 --- a/content/docs/examples/multicluster/icp/index.md +++ b/content/docs/examples/multicluster/icp/index.md @@ -7,7 +7,7 @@ keywords: [kubernetes,multicluster] This example demonstrates how to use Istio's multicluster feature to join two [IBM Cloud Private](https://www.ibm.com/cloud/private) clusters together, -using the [multicluster installation instructions](/docs/setup/kubernetes/multicluster-install/). +using the [VPN-based multicluster installation instructions](/docs/setup/kubernetes/multicluster-install/vpn/). ## Create the IBM Cloud Private Clusters @@ -132,7 +132,7 @@ across all nodes in the two IBM Cloud Private Clusters. ## Install Istio for multicluster -[Follow the multicluster installation steps](/docs/setup/kubernetes/multicluster-install/) to install and configure +[Follow the VPN-based multicluster installation steps](/docs/setup/kubernetes/multicluster-install/vpn/) to install and configure local Istio control plane and Istio remote on `cluster-1` and `cluster-2`. This example uses `cluster-1` as the local Istio control plane and `cluster-2` as the Istio remote. diff --git a/content/docs/examples/multicluster/iks-icp/index.md b/content/docs/examples/multicluster/iks-icp/index.md index de639a3852..c16fd2847e 100644 --- a/content/docs/examples/multicluster/iks-icp/index.md +++ b/content/docs/examples/multicluster/iks-icp/index.md @@ -7,7 +7,7 @@ keywords: [kubernetes,multicluster,hybrid] This example demonstrates how to use Istio's multicluster feature to join 1 [IBM Cloud Private](https://www.ibm.com/cloud/private) cluster and 1 [IBM Cloud Kubernetes Service](https://console.bluemix.net/docs/containers/container_index.html) cluster together, -using the [multicluster installation instructions](/docs/setup/kubernetes/multicluster-install/). +using the [VPN-based multicluster installation instructions](/docs/setup/kubernetes/multicluster-install/vpn/). ## Set up two clusters @@ -83,7 +83,7 @@ Since these two clusters are in isolated network environments we need to set up ## Install Istio for multicluster -[Follow the multicluster installation steps](/docs/setup/kubernetes/multicluster-install/) to install and configure +[Follow the VPN-based multicluster installation steps](/docs/setup/kubernetes/multicluster-install/vpn/) to install and configure the local Istio control plane and Istio remote on IBM Cloud Private and IBM Cloud Kubernetes Service. This example uses IBM Cloud Private as the Istio local control plane and IBM Cloud Kubernetes Service as the Istio remote. diff --git a/content/docs/setup/kubernetes/multicluster-install/_index.md b/content/docs/setup/kubernetes/multicluster-install/_index.md new file mode 100644 index 0000000000..52cf49878a --- /dev/null +++ b/content/docs/setup/kubernetes/multicluster-install/_index.md @@ -0,0 +1,7 @@ +--- +title: Multicluster Installation +description: Configure an Istio mesh across multiple kubernetes clusters. +weight: 60 +type: section-index +keywords: [kubernetes,multicluster,federation] +--- diff --git a/content/docs/setup/kubernetes/multicluster-install/gateways/index.md b/content/docs/setup/kubernetes/multicluster-install/gateways/index.md new file mode 100644 index 0000000000..cdd8767b26 --- /dev/null +++ b/content/docs/setup/kubernetes/multicluster-install/gateways/index.md @@ -0,0 +1,278 @@ +--- +title: Gateway connectivity +description: Install an Istio mesh across multiple Kubernetes clusters using Istio Gateway to reach remote pods. +weight: 2 +keywords: [kubernetes,multicluster,federation,gateway] +--- + +Instructions for installing an Istio mesh across multiple clusters when pods +in each cluster can only connect to remote gateway IPs. + +Instead of using a central Istio control plane to manage the mesh, +in this configuration each cluster has an **identical** Istio control plane +installation, each managing its own endpoints. +All of the clusters are under a shared administrative control for the purposes of +policy enforcement and security. + +A single Istio service mesh across the clusters is achieved by replicating +shared services and namespaces and using a common root CA in all of the clusters. +Cross-cluster communication occurs over Istio Gateways of the respective clusters. + +{{< image width="80%" ratio="36.01%" + link="./multicluster-with-gateways.svg" + caption="Istio mesh spanning multiple Kubernetes clusters using Istio Gateway to reach remote pods" + >}} + +## Prerequisites + +* Two or more Kubernetes clusters with **1.9 or newer**. + +* Authority to deploy the [Istio control plane using Helm](/docs/setup/kubernetes/helm-install/) +on **each** Kubernetes cluster. + +* The IP address of the Istio gateway in each cluster must be accessible + from every other cluster. + +* A **Root CA**. To enable mTLS communication across clusters, each cluster's + Citadel will be configured with intermediate CA credentials generated + by a shared root CA. + +## Deploy Istio control plane in each cluster + +1. Generate intermediate CA certs for each cluster's Citadel from your +organization's root CA. The shared root CA enables mTLS communication +across different clusters. + +1. Create a Kubernetes secret for your generated CA certs using a command similar + to the following: + + {{< text bash >}} + $ kubectl create secret generic cacerts -n istio-system \ + --from-file=path/to/intermediate/ca-cert.pem \ + --from-file=path/to/intermediate/ca-key.pem \ + --from-file=path/to/intermediate/root-cert.pem \ + --from-file=path/to/intermediate/cert-chain.pem + {{< /text >}} + +1. Install the Istio control plane using Helm: + + {{< text bash >}} + $ helm template install/kubernetes/helm/istio --name istio --namespace istio-system \ + --set multiCluster.connectUsingGateway=true \ + --set global.controlPlaneSecurityEnabled=true \ + --set security.selfSigned=false > $HOME/istio.yaml + $ kubectl create namespace istio-system + $ kubectl apply -f $HOME/istio.yaml + {{< /text >}} + +For further details and customization options, refer to the [Installation +with Helm](/docs/setup/kubernetes/helm-install/) instructions. + +## Enable mTLS globally + +Cross-cluster communication via the Istio gateway requires mTLS. +Therefore you must enable mTLS for all services in each cluster by +setting the global authentication policy to permissive or stricter +and setting a default destination rule to originate an mTLS connection +from every pod. + +For example, apply the following global configuration in each cluster: + +{{< text yaml >}} +apiVersion: authentication.istio.io/v1alpha1 +kind: MeshPolicy +metadata: + name: default +spec: + peers: + - mtls: + mode: PERMISSIVE +--- +apiVersion: networking.istio.io/v1alpha3 +kind: DestinationRule +metadata: + name: default +spec: + host: "*.local" + trafficPolicy: + tls: + mode: ISTIO_MUTUAL +{{< /text >}} + +## Replicate services and namespaces across clusters + +In order for pods in a cluster to be able to resolve DNS names of services +in remote clusters, you need to replicate every namespace and service +declaration from one cluster to every other cluster in the mesh. + +For example, the diagram above depicts two services `foo.ns1` in +`cluster1` and `bar.ns2` in `cluster2`. Lets say their respective +declarations are as follows: + +In `cluster1` (file: `cluster1-svc.yaml`) + +{{< text yaml >}} +apiVersion: v1 +kind: Namespace +metadata: + name: ns1 + labels: + istio-injection: enabled +--- +apiVersion: v1 +kind: Service +metadata: + name: foo + labels: + app: foo + namespace: ns1 +spec: + ports: + - port: 80 + targetPort: 80 + name: http + selector: + app: foo +{{< /text >}} + +In `cluster2` (file: `cluster2-svc.yaml`) + +{{< text yaml >}} +apiVersion: v1 +kind: Namespace +metadata: + name: ns2 + labels: + istio-injection: enabled +--- +apiVersion: v1 +kind: Service +metadata: + name: bar + labels: + app: bar + namespace: ns2 +spec: + ports: + - port: 80 + targetPort: 80 + name: http + - port: 9090 + targetPort: 9090 + name: tcp + selector: + app: bar +{{< /text >}} + +The namespaces and services need to be replicated in each cluster +to enable DNS resolution and connectivity. This can be done by simply applying +the configuration in `cluster1-svc.yaml` to `cluster2` and the configuration in +`cluster2-svc.yaml` to `cluster1`. + +In `cluster1`: + +{{< text bash >}} +$ kubectl apply -f cluster2-svc.yaml +{{< /text >}} + +In `cluster2`: + +{{< text bash >}} +$ kubectl apply -f cluster1-svc.yaml +{{< /text >}} + +## Route traffic for remote services to remote cluster gateway + +By replicating the remote services, you enabled DNS resolution for +them inside the application code running on a local cluster. However, network traffic +from the application will be sent to the cluster IP associated with the +Kubernetes service, instead of to the remote pods. + +To allow Envoy to route traffic to the remote services, you need to provide endpoints +for the remote service where traffic should be routed. In this case, that's +the remote cluster gateway. + +Continuing with the example from the previous section, +apply the following service entry in `cluster1`: + +{{< text yaml >}} +apiVersion: networking.istio.io/v1alpha3 +kind: ServiceEntry +metadata: + name: remote-endpoint-for-bar-in-cluster1 +spec: + hosts: + - bar.ns2.svc.cluster.local + ports: + # Add all ports from bar service + - number: 80 + name: http + protocol: HTTP + - number: 9090 + name: tcp + protocol: TCP + resolution: STATIC + location: MESH_INTERNAL + endpoints: + - address: + ports: + http: 15443 # Map everything to 15443 of ingress gateway + tcp: 15443 +{{< /text >}} + +This configuration will result in all traffic in `cluster1` for +`bar.ns2.svc.cluster.local` on any port to be routed to the endpoint +`:15443` over an mTLS connection. + +Apply a similar configuration for the `foo.ns1` namespace in `cluster2`: + +{{< text yaml >}} +apiVersion: networking.istio.io/v1alpha3 +kind: ServiceEntry +metadata: + name: remote-endpoint-for-foo-in-cluster2 +spec: + hosts: + - foo.ns1.svc.cluster.local + ports: + # Add all ports from foo service + - number: 80 + name: http + protocol: HTTP + resolution: STATIC + location: MESH_INTERNAL + endpoints: + - address: + ports: + http: 15443 # Map everything to 15443 of ingress gateway + tcp: 15443 +{{< /text >}} + +The gateway for port 15443 is a special SNI-aware Envoy that was +configured and installed as a result of the `--set multiCluster.connectUsingGateway=true` +helm option that you used when you installed Istio. Traffic entering port 15443 will +be load balanced among pods of the appropriate internal service of the target cluster (in this +case, `bar.ns2`). + +> Do not create a Gateway configuration for port 15443. + +## Summary + +Using Istio gateways, a common root CA, and replication of services, you configured +a single Istio service mesh across multiple Kubernetes clusters. +Although the above procedure involved a significant amount of manual work, +the entire process could theoretically be automated. The automation process +must do the following: + +1. Replicate Istio configuration, services, and namespaces across all clusters in the mesh. +1. Generate service entries for each replicated service, with endpoints containing the IP address + of gateways in every cluster that has pods for the service in question. + +Once configured this way, traffic can be transparently routed to remote clusters +without any application involvement. In fact, traffic for a given service can also +be routed to workloads in multiple clusters. For example, if you add pods to the +`bar.ns2` service in `cluster1`, traffic will be load +balanced across the local pods and the remote pods in `cluster2`. +The load will be skewed, however, as Istio in `cluster1` is unaware of the number +of pods for `bar.ns2` in `cluster2`. The remote Gateway looks like just another +single endpoint in the same cluster/zone. Zone-aware routing is currently unsupported. diff --git a/content/docs/setup/kubernetes/multicluster-install/gateways/multicluster-with-gateways.svg b/content/docs/setup/kubernetes/multicluster-install/gateways/multicluster-with-gateways.svg new file mode 100644 index 0000000000..7768e8c0f0 --- /dev/null +++ b/content/docs/setup/kubernetes/multicluster-install/gateways/multicluster-with-gateways.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/content/docs/setup/kubernetes/multicluster-install/index.md b/content/docs/setup/kubernetes/multicluster-install/vpn/index.md similarity index 96% rename from content/docs/setup/kubernetes/multicluster-install/index.md rename to content/docs/setup/kubernetes/multicluster-install/vpn/index.md index 1df11f8e8e..be1b02fa66 100644 --- a/content/docs/setup/kubernetes/multicluster-install/index.md +++ b/content/docs/setup/kubernetes/multicluster-install/vpn/index.md @@ -1,11 +1,25 @@ --- -title: Istio Multicluster -description: Install Istio with multicluster support. -weight: 60 -keywords: [kubernetes,multicluster] +title: VPN connectivity +description: Install an Istio mesh across multiple Kubernetes clusters with direct network access to remote pods. +weight: 5 +keywords: [kubernetes,multicluster,federation,vpn] +aliases: + - /docs/setup/kubernetes/multicluster-install --- -Instructions for the installation of Istio multicluster. +Instructions for installing an Istio mesh across multiple clusters when pods +in each cluster have direct network access to pods in other clusters. + +In this configuration, multiple Kubernetes control planes running +a remote configuration connect to a **single** Istio control plane. +Once one or more remote Kubernetes clusters are connected to the +Istio control plane, Envoy can then communicate with the **single** +control plane and form a mesh network across multiple clusters. + +{{< image width="80%" ratio="36.01%" + link="./multicluster-with-vpn.svg" + caption="Istio mesh spanning multiple Kubernetes clusters with direct network access to remote pods over VPN" + >}} ## Prerequisites @@ -26,15 +40,6 @@ across the multicluster environment and may not overlap. * Helm **2.7.2 or newer**. The use of Tiller is optional. -## Overview - -Multicluster functions by enabling Kubernetes control planes running -a remote configuration to connect to **one** Istio control plane. -Once one or more remote Kubernetes clusters are connected to the -Istio control plane, Envoy can then communicate with the **single** -Istio control plane and form a mesh network across multiple Kubernetes -clusters. - This guide describes how to install a multicluster Istio topology using the manifests and Helm charts provided within the Istio repository. diff --git a/content/docs/setup/kubernetes/multicluster-install/vpn/multicluster-with-vpn.svg b/content/docs/setup/kubernetes/multicluster-install/vpn/multicluster-with-vpn.svg new file mode 100644 index 0000000000..d0c701504a --- /dev/null +++ b/content/docs/setup/kubernetes/multicluster-install/vpn/multicluster-with-vpn.svg @@ -0,0 +1 @@ + \ No newline at end of file