Istioctl on remote clusters blog post (#11099)

* Istioctl proxy blog

* fix lint

* more lint

* update sample urls

* wording tweaks

* more tweaks

* Update content/en/blog/2022/istioctl-proxy/index.md

* Update content/en/blog/2022/istioctl-proxy/index.md

* Update content/en/blog/2022/istioctl-proxy/index.md

* Update content/en/blog/2022/istioctl-proxy/index.md

content/en/blog/2022/istioctl-proxy/index.md

@@ -0,0 +1,172 @@
+---
+title: Configuring istioctl for a remote cluster
+description: Using a proxy server to support istioctl commands in a mesh with an external control plane.
+publishdate: 2022-03-25
+attribution: Frank Budinsky (IBM)
+keywords: [istioctl, cli, external, remote, multicluster]
+---
+
+When using the `istioctl` CLI on a {{< gloss >}}remote cluster{{< /gloss >}} of an
+[external control plane](/docs/setup/install/external-controlplane/) or a [multicluster](/docs/setup/install/multicluster/)
+Istio deployment, some of the commands will not work by default. For example, `istioctl proxy-status` requires access to
+the `istiod` service to retrieve the status and configuration of the proxies it's managing. If you try running it on a
+remote cluster, you'll get an error message like this:
+
+{{< text bash >}}
+$ istioctl proxy-status
+Error: unable to find any Istiod instances
+{{< /text >}}
+
+Notice that the error message doesn't just say that it's unable to access the `istiod` service, it specifically mentions
+its inability to find `istiod` instances. This is because the `istioctl proxy-status` implementation needs to retrieve
+the sync status of not just any single `istiod` instance, but rather all of them. When there is more than one `istiod`
+instance (replica) running, each instance is only connected to a subset of the service proxies running in the mesh.
+The `istioctl` command needs to return the status for the entire mesh, not just the subset managed by one of the instances.
+
+In an ordinary Istio installation where the `istiod` service is running locally on the cluster
+(i.e., a {{< gloss >}}primary cluster{{< /gloss >}}), the command is implemented by simply finding all of the running
+`istiod` pods, calling each one in turn, and then aggregating the result before returning it to the user.
+
+{{< image width="75%"
+    link="istioctl-primary-cluster.svg"
+    caption="CLI with local access to istiod pods"
+    >}}
+
+When using a remote cluster, on the other hand, this is not possible since the `istiod` instances are running outside
+of the mesh cluster and not accessible to the mesh user. The instances may not even be deployed using pods on a Kubernetes
+cluster.
+
+Fortunately, `istioctl` provides a configuration option to address this issue.
+You can configure `istioctl` with the address of an external proxy service that will have access to the
+`istiod` instances. Unlike an ordinary load-balancer service, which would delegate incoming requests to one of the
+instances, this proxy service must instead delegate to all of the `istiod` instances, aggregate the responses,
+and then return the combined result.
+
+If the external proxy service is, in fact, running on another Kubernetes cluster, the proxy implementation code
+can be very similar to the implementation code that `istioctl` runs in the primary cluster case, i.e., find all of the
+running `istiod` pods, call each one in turn, and then aggregate the result.
+
+{{< image width="75%"
+    link="istioctl-remote-cluster.svg"
+    caption="CLI without local access to istiod pods"
+    >}}
+
+An Istio Ecosystem project that includes an implementation of such an `istioctl` proxy server can be found
+[here](https://github.com/istio-ecosystem/istioctl-proxy-sample). To try it out, you'll need two clusters, one of which is
+configured as a remote cluster using a control plane installed in the other cluster.
+
+## Install Istio with a remote cluster topology
+
+To demonstrate `istioctl` working on a remote cluster, we'll start by using the
+[external control plane install instructions](/docs/setup/install/external-controlplane/)
+to set up a single remote cluster mesh with an external control plane running in a separate external cluster.
+
+After completing the installation, we should have two environment variables, `CTX_REMOTE_CLUSTER` and `CTX_EXTERNAL_CLUSTER`,
+containing the context names of the remote (mesh) and external (control plane) clusters, respectively.
+
+We should also have the `helloworld` and `sleep` samples running in the mesh, i.e., on the remote cluster:
+
+{{< text bash >}}
+$ kubectl get pod -n sample --context="${CTX_REMOTE_CLUSTER}"
+NAME                             READY   STATUS    RESTARTS   AGE
+helloworld-v1-776f57d5f6-tmpkd   2/2     Running   0          10s
+sleep-557747455f-v627d           2/2     Running   0          9s
+{{< /text >}}
+
+Notice that if you try to run `istioctl proxy-status` in the remote cluster, you will see the error message
+described earlier:
+
+{{< text bash >}}
+$ istioctl proxy-status --context="${CTX_REMOTE_CLUSTER}"
+Error: unable to find any Istiod instances
+{{< /text >}}
+
+## Configure istioctl to use the sample proxy service
+
+To configure `istioctl`, we first need to deploy  the proxy service next to the running `istiod` pods.
+In our installation, we've deployed the control plane in the `external-istiod` namespace, so we start the proxy
+service on the external cluster using the following command:
+
+{{< text bash >}}
+$ kubectl apply -n external-istiod --context="${CTX_EXTERNAL_CLUSTER}" \
+    -f https://raw.githubusercontent.com/istio-ecosystem/istioctl-proxy-sample/main/istioctl-proxy.yaml
+service/istioctl-proxy created
+serviceaccount/istioctl-proxy created
+secret/jwt-cert-key-secret created
+deployment.apps/istioctl-proxy created
+role.rbac.authorization.k8s.io/istioctl-proxy-role created
+rolebinding.rbac.authorization.k8s.io/istioctl-proxy-role created
+{{< /text >}}
+
+You can run the following command to confirm that the `istioctl-proxy` service is running next to `istiod`:
+
+{{< text bash >}}
+$ kubectl get po -n external-istiod --context="${CTX_EXTERNAL_CLUSTER}"
+NAME                              READY   STATUS    RESTARTS   AGE
+istioctl-proxy-664bcc596f-9q8px   1/1     Running   0          15s
+istiod-666fb6694d-jklkt           1/1     Running   0          5m31s
+{{< /text >}}
+
+The proxy service is a gRPC server that is serving on port 9090:
+
+{{< text bash >}}
+$ kubectl get svc istioctl-proxy -n external-istiod --context="${CTX_EXTERNAL_CLUSTER}"
+NAME             TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
+istioctl-proxy   ClusterIP   172.21.127.192   <none>        9090/TCP   11m
+{{< /text >}}
+
+Before we can use it, however, we need to expose it outside of the external cluster.
+There are many ways to do that, depending on the deployment environment. In our setup, we have an ingress gateway
+running on the external cluster, so we could update it to also expose port 9090, update the associated virtual service
+to direct port 9090 requests to the proxy service, and then configure `istioctl` to use the gateway address for the proxy
+service. This would be a "proper" approach.
+
+However, since this is just a simple demonstration where we have access to both clusters, we will simply `port-forward`
+the proxy service to `localhost`:
+
+{{< text bash >}}
+$ kubectl port-forward -n external-istiod service/istioctl-proxy 9090:9090 --context="${CTX_EXTERNAL_CLUSTER}"
+{{< /text >}}
+
+We now configure `istioctl` to use `localhost:9090` to access the proxy by setting the `ISTIOCTL_XDS_ADDRESS` environment
+variable:
+
+{{< text bash >}}
+$ export ISTIOCTL_XDS_ADDRESS=localhost:9090
+$ export ISTIOCTL_ISTIONAMESPACE=external-istiod
+$ export ISTIOCTL_PREFER_EXPERIMENTAL=true
+{{< /text >}}
+
+Because our control plane is running in the `external-istiod` namespace, instead of the default `istio-system`, we also
+need to set the `ISTIOCTL_ISTIONAMESPACE` environment variable.
+
+Setting `ISTIOCTL_PREFER_EXPERIMENTAL` is optional. It instructs `istioctl` to redirect `istioctl command` calls to
+an experimental equivalent, `istioctl x command`, for any `command` that has both a stable and experimental implementation.
+In our case we need to use `istioctl x proxy-status`, the version that implements the proxy delegation feature.
+
+## Run the istioctl proxy-status command
+
+Now that we're finished configuring `istioctl` we can try it out by running the `proxy-status` command again:
+
+{{< text bash >}}
+$ istioctl proxy-status --context="${CTX_REMOTE_CLUSTER}"
+NAME                                                      CDS        LDS        EDS        RDS        ISTIOD         VERSION
+helloworld-v1-776f57d5f6-tmpkd.sample                     SYNCED     SYNCED     SYNCED     SYNCED     <external>     1.12.1
+istio-ingressgateway-75bfd5668f-lggn4.external-istiod     SYNCED     SYNCED     SYNCED     SYNCED     <external>     1.12.1
+sleep-557747455f-v627d.sample                             SYNCED     SYNCED     SYNCED     SYNCED     <external>     1.12.1
+{{< /text >}}
+
+As you can see, this time it correctly displays the sync status of all the services running in the mesh. Notice that the
+`ISTIOD` column returns the generic value `<external>`, instead of the instance name (e.g., `istiod-666fb6694d-jklkt`)
+that would be displayed if the pod was running locally. In this case, this detail is not available, or needed, by the
+mesh user. It's only available on the external cluster for the mesh operator to see.
+
+## Summary
+
+In this article, we used a [sample proxy server](https://github.com/istio-ecosystem/istioctl-proxy-sample) to configure `istioctl` to
+work with an [external control plane installation](/docs/setup/install/external-controlplane/).
+We've seen how some of the `istioctl` CLI commands don't work out of the box on a remote cluster managed
+by an external control plane. Commands such as `istioctl proxy-status`, among others, need access to the `istiod` service
+instances managing the mesh, which are unavailable when the control plane is running outside of the mesh cluster.
+To address this issue, `istioctl` was configured to delegate to a proxy server, running along side the external control
+plane, which accesses the `istiod` instances on its behalf.