304 lines
7.0 KiB
Markdown
304 lines
7.0 KiB
Markdown
# Multi-cluster Ingress
|
|
|
|
Users can use [MultiClusterIngress API](https://github.com/karmada-io/karmada/blob/master/pkg/apis/networking/v1alpha1/ingress_types.go) provided in Karmada to import external traffic to services in the member clusters.
|
|
|
|
## Prerequisites
|
|
|
|
### Karmada has been installed
|
|
|
|
We can install Karmada by referring to [quick-start](https://github.com/karmada-io/karmada#quick-start), or directly run `hack/local-up-karmada.sh` script which is also used to run our E2E cases.
|
|
|
|
### Cluster Network
|
|
|
|
Currently, we need to use the [MCS](https://github.com/karmada-io/karmada/blob/master/docs/multi-cluster-service.md#the-serviceexport-and-serviceimport-crds-have-been-installed) feature to import external traffic.
|
|
|
|
So we need to ensure that the container networks between the **host cluster** and member clusters are connected. The **host cluster** indicates the cluster where the Karmada control plane is deployed.
|
|
|
|
- If you use the `hack/local-up-karmada.sh` script to deploy Karmada, Karmada will have three member clusters, and the container networks between the **host cluster**, `member1` and `member2` are connected.
|
|
- You can use `Submariner` or other related open source projects to connected networks between clusters.
|
|
|
|
## Example
|
|
|
|
### Step 1: Deploy ingress-nginx on the host cluster
|
|
|
|
We use [multi-cluster-ingress-nginx](https://github.com/karmada-io/multi-cluster-ingress-nginx) as the demo for demonstration. We've made some changes based on the latest version(controller-v1.1.1) of [ingress-nginx](https://github.com/kubernetes/ingress-nginx).
|
|
|
|
#### Download code
|
|
|
|
```
|
|
// for HTTPS
|
|
git clone https://github.com/karmada-io/multi-cluster-ingress-nginx.git
|
|
// for SSH
|
|
git clone git@github.com:karmada-io/multi-cluster-ingress-nginx.git
|
|
```
|
|
|
|
#### Build and deploy ingress-nginx
|
|
|
|
Using the existing `karmada-host` kind cluster to build and deploy the ingress controller.
|
|
|
|
```
|
|
export KIND_CLUSTER_NAME=karmada-host
|
|
kubectl config use-context karmada-host
|
|
make dev-env
|
|
```
|
|
|
|
#### Apply kubeconfig secret
|
|
|
|
Create a secret that contains the `karmada-apiserver` authentication credential in the following format:
|
|
|
|
```yaml
|
|
# karmada-kubeconfig-secret.yaml
|
|
apiVersion: v1
|
|
data:
|
|
kubeconfig: {data} # Base64-encoded
|
|
kind: Secret
|
|
metadata:
|
|
name: kubeconfig
|
|
namespace: ingress-nginx
|
|
type: Opaque
|
|
```
|
|
|
|
You can get the authentication credential from the `/root/.kube/karmada.config` file, or use command:
|
|
|
|
```
|
|
kubectl -n karmada-system get secret kubeconfig -oyaml | grep kubeconfig | sed -n '1p' | awk '{print $2}'
|
|
```
|
|
|
|
Then apply yaml:
|
|
|
|
```
|
|
kubectl apply -f karmada-kubeconfig-secret.yaml
|
|
```
|
|
|
|
#### Edit ingress-nginx-controller deployment
|
|
|
|
We want `nginx-ingress-controller` to access `karmada-apiserver` to listen to changes in resources(such as multiclusteringress, endpointslices, and service). Therefore, we need to mount the authentication credential of `karmada-apiserver` to the `nginx-ingress-controller`.
|
|
|
|
```
|
|
kubectl -n ingress-nginx edit deployment ingress-nginx-controller
|
|
```
|
|
|
|
Edit as follows:
|
|
|
|
```yaml
|
|
apiVersion: apps/v1
|
|
kind: Deployment
|
|
metadata:
|
|
...
|
|
spec:
|
|
...
|
|
template:
|
|
spec:
|
|
containers:
|
|
- args:
|
|
- /nginx-ingress-controller
|
|
- --karmada-kubeconfig=/etc/kubeconfig # new line
|
|
...
|
|
volumeMounts:
|
|
...
|
|
- mountPath: /etc/kubeconfig # new line
|
|
name: kubeconfig # new line
|
|
subPath: kubeconfig # new line
|
|
volumes:
|
|
...
|
|
- name: kubeconfig # new line
|
|
secret: # new line
|
|
secretName: kubeconfig # new line
|
|
```
|
|
|
|
### Step 2: Use the MCS feature to discovery service
|
|
|
|
#### Install ServiceExport and ServiceImport CRDs
|
|
|
|
Refer to [here](https://github.com/karmada-io/karmada/blob/master/docs/multi-cluster-service.md#the-serviceexport-and-serviceimport-crds-have-been-installed).
|
|
|
|
#### Deploy web on member1 cluster
|
|
|
|
deploy.yaml:
|
|
|
|
<details>
|
|
|
|
<summary>unfold me to see the yaml</summary>
|
|
|
|
```yaml
|
|
apiVersion: apps/v1
|
|
kind: Deployment
|
|
metadata:
|
|
name: web
|
|
spec:
|
|
replicas: 1
|
|
selector:
|
|
matchLabels:
|
|
app: web
|
|
template:
|
|
metadata:
|
|
labels:
|
|
app: web
|
|
spec:
|
|
containers:
|
|
- name: hello-app
|
|
image: gcr.io/google-samples/hello-app:1.0
|
|
ports:
|
|
- containerPort: 8080
|
|
protocol: TCP
|
|
---
|
|
apiVersion: v1
|
|
kind: Service
|
|
metadata:
|
|
name: web
|
|
spec:
|
|
ports:
|
|
- port: 81
|
|
targetPort: 8080
|
|
selector:
|
|
app: web
|
|
---
|
|
apiVersion: policy.karmada.io/v1alpha1
|
|
kind: PropagationPolicy
|
|
metadata:
|
|
name: mcs-workload
|
|
spec:
|
|
resourceSelectors:
|
|
- apiVersion: apps/v1
|
|
kind: Deployment
|
|
name: web
|
|
- apiVersion: v1
|
|
kind: Service
|
|
name: web
|
|
placement:
|
|
clusterAffinity:
|
|
clusterNames:
|
|
- member1
|
|
```
|
|
|
|
</details>
|
|
|
|
```
|
|
kubectl --context karmada-apiserver apply -f deploy.yaml
|
|
```
|
|
|
|
#### Export web service from member1 cluster
|
|
|
|
service_export.yaml:
|
|
|
|
<details>
|
|
|
|
<summary>unfold me to see the yaml</summary>
|
|
|
|
```yaml
|
|
apiVersion: multicluster.x-k8s.io/v1alpha1
|
|
kind: ServiceExport
|
|
metadata:
|
|
name: web
|
|
---
|
|
apiVersion: policy.karmada.io/v1alpha1
|
|
kind: PropagationPolicy
|
|
metadata:
|
|
name: web-export-policy
|
|
spec:
|
|
resourceSelectors:
|
|
- apiVersion: multicluster.x-k8s.io/v1alpha1
|
|
kind: ServiceExport
|
|
name: web
|
|
placement:
|
|
clusterAffinity:
|
|
clusterNames:
|
|
- member1
|
|
```
|
|
|
|
</details>
|
|
|
|
```
|
|
kubectl --context karmada-apiserver apply -f service_export.yaml
|
|
```
|
|
|
|
#### Import web service to member2 cluster
|
|
|
|
service_import.yaml:
|
|
|
|
<details>
|
|
|
|
<summary>unfold me to see the yaml</summary>
|
|
|
|
```yaml
|
|
apiVersion: multicluster.x-k8s.io/v1alpha1
|
|
kind: ServiceImport
|
|
metadata:
|
|
name: web
|
|
spec:
|
|
type: ClusterSetIP
|
|
ports:
|
|
- port: 81
|
|
protocol: TCP
|
|
---
|
|
apiVersion: policy.karmada.io/v1alpha1
|
|
kind: PropagationPolicy
|
|
metadata:
|
|
name: web-import-policy
|
|
spec:
|
|
resourceSelectors:
|
|
- apiVersion: multicluster.x-k8s.io/v1alpha1
|
|
kind: ServiceImport
|
|
name: web
|
|
placement:
|
|
clusterAffinity:
|
|
clusterNames:
|
|
- member2
|
|
```
|
|
|
|
</details>
|
|
|
|
```
|
|
kubectl --context karmada-apiserver apply -f service_import.yaml
|
|
```
|
|
|
|
### Step 3: Deploy multiclusteringress on karmada-controlplane
|
|
|
|
mci-web.yaml:
|
|
|
|
<details>
|
|
|
|
<summary>unfold me to see the yaml</summary>
|
|
|
|
```yaml
|
|
apiVersion: networking.karmada.io/v1alpha1
|
|
kind: MultiClusterIngress
|
|
metadata:
|
|
name: demo-localhost
|
|
namespace: default
|
|
spec:
|
|
ingressClassName: nginx
|
|
rules:
|
|
- host: demo.localdev.me
|
|
http:
|
|
paths:
|
|
- backend:
|
|
service:
|
|
name: web
|
|
port:
|
|
number: 81
|
|
path: /web
|
|
pathType: Prefix
|
|
```
|
|
|
|
</details>
|
|
|
|
```
|
|
kubectl --context karmada-apiserver apply -f
|
|
```
|
|
|
|
### Step 4: Local testing
|
|
|
|
Let's forward a local port to the ingress controller:
|
|
|
|
```
|
|
kubectl port-forward --namespace=ingress-nginx service/ingress-nginx-controller 8080:80
|
|
```
|
|
|
|
At this point, if you access http://demo.localdev.me:8080/web/, you should see an HTML page telling you:
|
|
|
|
```html
|
|
Hello, world!
|
|
Version: 1.0.0
|
|
Hostname: web-xxx-xxx
|
|
``` |