Update descriptions
Signed-off-by: RainbowMango <qdurenhongcai@gmail.com>
This commit is contained in:
RainbowMango
parent
cb1e7441bf
commit
b5cdb6bb58
|
|
@ -4,72 +4,221 @@ Follow the [Regular Upgrading Process](./README.md).
|
|||
|
||||
## Upgrading Notable Changes
|
||||
|
||||
### APIChanges
|
||||
### Introduced `karmada-aggregated-apiserver` component
|
||||
|
||||
Previously, we used CRD to extend the [Cluster API](https://github.com/karmada-io/karmada/blob/v0.10.1/charts/_crds/bases/cluster.karmada.io_clusters.yaml), however, in the version v1.0, we change to use [API Aggregation](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/apiserver-aggregation/)(AA) to extend the Cluster API.
|
||||
In the releases before v1.0.0, we are using CRD to extend the
|
||||
[Cluster API](https://github.com/karmada-io/karmada/tree/24f586062e0cd7c9d8e6911e52ce399106f489aa/pkg/apis/cluster),
|
||||
and starts v1.0.0 we use
|
||||
[API Aggregation](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/apiserver-aggregation/)(AA) to
|
||||
extend it.
|
||||
|
||||
Based on the above change, perform the following operations during the upgrade:
|
||||
|
||||
1. Chang the replicas of `karmada-apiserver` to 0.
|
||||
#### Step 1: Stop `karmada-apiserver`
|
||||
|
||||
2. Operate etcd to delete cluster crd data.
|
||||
You can stop `karmada-apiserver` by updating its replica to `0`.
|
||||
|
||||
```
|
||||
etcdctl --cert="/etc/kubernetes/pki/etcd/karmada.crt" --key="/etc/kubernetes/pki/etcd/karmada.key" --cacert="/etc/kubernetes/pki/etcd/server-ca.crt" del /registry/apiextensions.k8s.io/customresourcedefinitions/clusters.cluster.karmada.io
|
||||
```
|
||||
#### Step 2: Remove Cluster CRD from ETCD
|
||||
|
||||
3. To avoid [CA Reusage and Conflicts](https://kubernetes.io/docs/tasks/extend-kubernetes/configure-aggregation-layer/#ca-reusage-and-conflicts), create CA signer and sign a certificate to enable the aggregation layer.
|
||||
|
||||
Update `karmada-cert-secret` secret in `karmada-system` namespace:
|
||||
|
||||
```diff
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: karmada-cert-secret
|
||||
namespace: karmada-system
|
||||
type: Opaque
|
||||
data:
|
||||
...
|
||||
+ front-proxy-ca.crt: |
|
||||
+ {{front_proxy_ca_crt}}
|
||||
+ front-proxy-client.crt: |
|
||||
+ {{front_proxy_client_crt}}
|
||||
+ front-proxy-client.key: |
|
||||
+ {{front_proxy_client_key}}
|
||||
```
|
||||
|
||||
And update `karmada-apiserver` deployment's container commend:
|
||||
|
||||
```diff
|
||||
- - --proxy-client-cert-file=/etc/kubernetes/pki/karmada.crt
|
||||
- - --proxy-client-key-file=/etc/kubernetes/pki/karmada.key
|
||||
+ - --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt
|
||||
+ - --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key
|
||||
- - --requestheader-client-ca-file=/etc/kubernetes/pki/server-ca.crt
|
||||
+ - --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt
|
||||
```
|
||||
|
||||
After the updation, restore the replicas of `karmada-apiserver` instances.
|
||||
|
||||
4. Deploy `karmada-aggregated-apiserver`:
|
||||
|
||||
```
|
||||
make image-karmada-aggregated-apiserver
|
||||
|
||||
kubectl --kubeconfig /root/.kube/karmada.config --context karmada-host apply -f artifacts/deploy/karmada-aggregated-apiserver.yaml
|
||||
|
||||
kubectl --kubeconfig /root/.kube/karmada.config --context karmada-apiserver apply -f artifacts/deploy/apiservice.yaml
|
||||
```
|
||||
|
||||
###karmada-agent
|
||||
|
||||
Due to [add unfied auth controller](https://github.com/karmada-io/karmada/pull/1104), we need to apply `karmada-agent` ClusterRole:
|
||||
Remove the `Cluster CRD` from ETCD directly by running the following command.
|
||||
|
||||
```
|
||||
kubectl apply -f artifacts/agent/clusterrole.yaml
|
||||
etcdctl --cert="/etc/kubernetes/pki/etcd/karmada.crt" \
|
||||
--key="/etc/kubernetes/pki/etcd/karmada.key" \
|
||||
--cacert="/etc/kubernetes/pki/etcd/server-ca.crt" \
|
||||
del /registry/apiextensions.k8s.io/customresourcedefinitions/clusters.cluster.karmada.io
|
||||
```
|
||||
|
||||
### Other
|
||||
> Note: This command only removed the `CRD` resource, all the `CR` (Cluster objects) not changed.
|
||||
> That's the reason why we don't remove CRD by `karmada-apiserver`.
|
||||
|
||||
If you need to use [MCS](https://github.com/karmada-io/karmada/blob/master/docs/multi-cluster-service.md) feature, we need to upgrade the version of member cluster's kube-apiserver to v1.21.x. For details about the upgrade reasons, see [comment](https://github.com/karmada-io/karmada/pull/1107#issuecomment-997159415).
|
||||
#### Step 3: Prepare the certificate for the `karmada-aggregated-apiserver`
|
||||
|
||||
To avoid [CA Reusage and Conflicts](https://kubernetes.io/docs/tasks/extend-kubernetes/configure-aggregation-layer/#ca-reusage-and-conflicts),
|
||||
create CA signer and sign a certificate to enable the aggregation layer.
|
||||
|
||||
Update `karmada-cert-secret` secret in `karmada-system` namespace:
|
||||
|
||||
```diff
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: karmada-cert-secret
|
||||
namespace: karmada-system
|
||||
type: Opaque
|
||||
data:
|
||||
...
|
||||
+ front-proxy-ca.crt: |
|
||||
+ {{front_proxy_ca_crt}}
|
||||
+ front-proxy-client.crt: |
|
||||
+ {{front_proxy_client_crt}}
|
||||
+ front-proxy-client.key: |
|
||||
+ {{front_proxy_client_key}}
|
||||
```
|
||||
|
||||
Then update `karmada-apiserver` deployment's container command:
|
||||
|
||||
```diff
|
||||
- - --proxy-client-cert-file=/etc/kubernetes/pki/karmada.crt
|
||||
- - --proxy-client-key-file=/etc/kubernetes/pki/karmada.key
|
||||
+ - --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt
|
||||
+ - --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key
|
||||
- - --requestheader-client-ca-file=/etc/kubernetes/pki/server-ca.crt
|
||||
+ - --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt
|
||||
```
|
||||
|
||||
After the update, restore the replicas of `karmada-apiserver` instances.
|
||||
|
||||
#### Step 4: Deploy `karmada-aggregated-apiserver`:
|
||||
|
||||
Deploy `karmada-aggregated-apiserver` instance to your `host cluster` by following manifests:
|
||||
<details>
|
||||
<summary>unfold me to see the yaml</summary>
|
||||
|
||||
```yaml
|
||||
---
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: karmada-aggregated-apiserver
|
||||
namespace: karmada-system
|
||||
labels:
|
||||
app: karmada-aggregated-apiserver
|
||||
apiserver: "true"
|
||||
spec:
|
||||
selector:
|
||||
matchLabels:
|
||||
app: karmada-aggregated-apiserver
|
||||
apiserver: "true"
|
||||
replicas: 1
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: karmada-aggregated-apiserver
|
||||
apiserver: "true"
|
||||
spec:
|
||||
automountServiceAccountToken: false
|
||||
containers:
|
||||
- name: karmada-aggregated-apiserver
|
||||
image: swr.ap-southeast-1.myhuaweicloud.com/karmada/karmada-aggregated-apiserver:v1.0.0
|
||||
imagePullPolicy: IfNotPresent
|
||||
volumeMounts:
|
||||
- name: k8s-certs
|
||||
mountPath: /etc/kubernetes/pki
|
||||
readOnly: true
|
||||
- name: kubeconfig
|
||||
subPath: kubeconfig
|
||||
mountPath: /etc/kubeconfig
|
||||
command:
|
||||
- /bin/karmada-aggregated-apiserver
|
||||
- --kubeconfig=/etc/kubeconfig
|
||||
- --authentication-kubeconfig=/etc/kubeconfig
|
||||
- --authorization-kubeconfig=/etc/kubeconfig
|
||||
- --karmada-config=/etc/kubeconfig
|
||||
- --etcd-servers=https://etcd-client.karmada-system.svc.cluster.local:2379
|
||||
- --etcd-cafile=/etc/kubernetes/pki/server-ca.crt
|
||||
- --etcd-certfile=/etc/kubernetes/pki/karmada.crt
|
||||
- --etcd-keyfile=/etc/kubernetes/pki/karmada.key
|
||||
- --tls-cert-file=/etc/kubernetes/pki/karmada.crt
|
||||
- --tls-private-key-file=/etc/kubernetes/pki/karmada.key
|
||||
- --audit-log-path=-
|
||||
- --feature-gates=APIPriorityAndFairness=false
|
||||
- --audit-log-maxage=0
|
||||
- --audit-log-maxbackup=0
|
||||
resources:
|
||||
requests:
|
||||
cpu: 100m
|
||||
volumes:
|
||||
- name: k8s-certs
|
||||
secret:
|
||||
secretName: karmada-cert-secret
|
||||
- name: kubeconfig
|
||||
secret:
|
||||
secretName: kubeconfig
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: karmada-aggregated-apiserver
|
||||
namespace: karmada-system
|
||||
labels:
|
||||
app: karmada-aggregated-apiserver
|
||||
apiserver: "true"
|
||||
spec:
|
||||
ports:
|
||||
- port: 443
|
||||
protocol: TCP
|
||||
targetPort: 443
|
||||
selector:
|
||||
app: karmada-aggregated-apiserver
|
||||
```
|
||||
</details>
|
||||
|
||||
Then, deploy `APIService` to `karmada-apiserver` by following manifests.
|
||||
|
||||
<details>
|
||||
<summary>unfold me to see the yaml</summary>
|
||||
|
||||
```yaml
|
||||
apiVersion: apiregistration.k8s.io/v1
|
||||
kind: APIService
|
||||
metadata:
|
||||
name: v1alpha1.cluster.karmada.io
|
||||
labels:
|
||||
app: karmada-aggregated-apiserver
|
||||
apiserver: "true"
|
||||
spec:
|
||||
insecureSkipTLSVerify: true
|
||||
group: cluster.karmada.io
|
||||
groupPriorityMinimum: 2000
|
||||
service:
|
||||
name: karmada-aggregated-apiserver
|
||||
namespace: karmada-system
|
||||
version: v1alpha1
|
||||
versionPriority: 10
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: karmada-aggregated-apiserver
|
||||
namespace: karmada-system
|
||||
spec:
|
||||
type: ExternalName
|
||||
externalName: karmada-aggregated-apiserver.karmada-system.svc.cluster.local
|
||||
```
|
||||
|
||||
</details>
|
||||
|
||||
#### Step 5: check clusters status
|
||||
|
||||
If everything goes well, you can see all your clusters just as before the upgrading.
|
||||
```yaml
|
||||
kubectl get clusters
|
||||
```
|
||||
|
||||
### `karmada-agent` requires an extra `impersonate` verb
|
||||
|
||||
In order to proxy user's request, the `karmada-agent` now request an extra `impersonate` verb.
|
||||
Please check the `ClusterRole` configuration or apply the following manifest.
|
||||
|
||||
```yaml
|
||||
apiVersion: rbac.authorization.k8s.io/v1
|
||||
kind: ClusterRole
|
||||
metadata:
|
||||
name: karmada-agent
|
||||
rules:
|
||||
- apiGroups: ['*']
|
||||
resources: ['*']
|
||||
verbs: ['*']
|
||||
- nonResourceURLs: ['*']
|
||||
verbs: ["get"]
|
||||
|
||||
```
|
||||
|
||||
### MCS feature now supports `Kubernetes v1.21+`
|
||||
|
||||
Since the `discovery.k8s.io/v1beta1` of `EndpointSlices` has been deprecated in favor of `discovery.k8s.io/v1`, in
|
||||
[Kubernetes v1.21](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.21.md), Karmada adopt
|
||||
this change at release v1.0.0.
|
||||
Now the [MCS](https://github.com/karmada-io/karmada/blob/master/docs/multi-cluster-service.md) feature requires
|
||||
member cluster version no less than v1.21.
|
||||
|
|
|
|||
Loading…
Reference in New Issue