Step-by-step installation of binary high-availability `karmada` cluster. # Installing Karmada cluster ## Prerequisites #### server 3 servers required. E.g. ```shell +---------------+-----------------+-----------------+ | HostName | Host IP | Public IP | +---------------+-----------------+-----------------+ | karmada-01 | 172.31.209.245 | 47.242.88.82 | +---------------+-----------------+-----------------+ | karmada-02 | 172.31.209.246 | | +---------------+-----------------+-----------------+ | karmada-03 | 172.31.209.247 | | +---------------+-----------------+-----------------+ ``` > Public IP is not required. It is used to download some `karmada` dependent components from the public network and connect to `karmada` ApiServer through the public network #### hosts parsing Execute operations at `karmada-01` `karmada-02` `karmada-03`. ```bash vi /etc/hosts 172.31.209.245 karmada-01 172.31.209.246 karmada-02 172.31.209.247 karmada-03 ``` #### environment `karmada-01` requires the following environment. **Golang**: Compile the karmada binary **GCC**: Compile nginx (ignore if using cloud load balancing) ## Compile and download binaries Execute operations at `karmada-01`. #### kubernetes binaries Download the `kubernetes` binary package. ```bash wget https://dl.k8s.io/v1.23.3/kubernetes-server-linux-amd64.tar.gz tar -zxvf kubernetes-server-linux-amd64.tar.gz cd /root/kubernetes/server/bin mv kube-apiserver kube-controller-manager kubectl /usr/local/sbin/ ``` #### etcd binaries Download the `etcd` binary package. ```bash wget https://github.com/etcd-io/etcd/releases/download/v3.5.1/etcd-v3.5.1-linux-amd64.tar.gz tar -zxvf etcd-v3.5.1-linux-amd64.tar.gz cd etcd-v3.5.1-linux-amd64/ cp etcdctl etcd /usr/local/sbin/ ``` #### karmada binaries Compile the `karmada` binary from source. ```bash git clone https://github.com/karmada-io/karmada cd karmada make karmada-aggregated-apiserver make karmada-controller-manager make karmada-scheduler make karmada-webhook mv karmada-aggregated-apiserver karmada-controller-manager karmada-scheduler karmada-webhook /usr/local/sbin/ ``` #### nginx binaries Compile the `nginx` binary from source. ```bash wget http://nginx.org/download/nginx-1.21.6.tar.gz tar -zxvf nginx-1.21.6.tar.gz cd nginx-1.21.6 ./configure --with-stream --without-http --prefix=/usr/local/karmada-nginx --without-http_uwsgi_module --without-http_scgi_module --without-http_fastcgi_module make && make install mv /usr/local/karmada-nginx/sbin/nginx /usr/local/karmada-nginx/sbin/karmada-nginx ``` #### Distribute binaries Upload the binary file to the `karmada-02` `karmada-03 ` server ```bash scp /usr/local/sbin/* karmada-02:/usr/local/sbin/ scp /usr/local/sbin/* karmada-03:/usr/local/sbin/ ``` ## Generate certificate Generated using the `openssl` command. Note yes `DNS` and `IP` when generating the certificate. Execute operations at `karmada-01`. #### create a temporary directory for certificates ```bash mkdir certs cd certs ``` #### Create root certificate valid for 10 years ```bash openssl genrsa -out ca.key 2048 openssl req -x509 -new -nodes -key ca.key -subj "/C=CN/ST=Guangdong/L=Guangzhou/O=karmada/OU=System/CN=karmada" -days 3650 -out ca.crt ``` #### Create etcd certificate create `etcd server ` certificate ```bash openssl genrsa -out etcd-server.key 2048 openssl req -new -nodes -sha256 -subj "/C=CN/ST=Guangdong/L=Guangzhou/O=karmada/OU=System/CN=karmada-etcd" -key etcd-server.key -out etcd-server.csr openssl x509 -req -days 3650 \ -extfile <(printf "keyUsage=critical,Digital Signature, Key Encipherment\nextendedKeyUsage=serverAuth,clientAuth\nauthorityKeyIdentifier=keyid,issuer\nsubjectAltName=IP:172.31.209.245,IP:172.31.209.246,IP:172.31.209.247,IP:127.0.0.1,DNS:localhost") \ -sha256 -CA ca.crt -CAkey ca.key -set_serial 01 -in etcd-server.csr -out etcd-server.crt ``` create `etcd peer ` certificate ```bash openssl genrsa -out etcd-peer.key 2048 openssl req -new -nodes -sha256 -subj "/C=CN/ST=Guangdong/L=Guangzhou/O=karmada/OU=System/CN=karmada-etcd-peer" -key etcd-peer.key -out etcd-peer.csr openssl x509 -req -days 3650 \ -extfile <(printf "keyUsage=critical,Digital Signature, Key Encipherment\nextendedKeyUsage=serverAuth,clientAuth\nauthorityKeyIdentifier=keyid,issuer\nsubjectAltName=IP:172.31.209.245,IP:172.31.209.246,IP:172.31.209.247,IP:127.0.0.1,DNS:localhost") \ -sha256 -CA ca.crt -CAkey ca.key -set_serial 01 -in etcd-peer.csr -out etcd-peer.crt ``` create `etcd client ` certificate ```bash openssl genrsa -out karmada-etcd-client.key 2048 openssl req -new -nodes -sha256 -subj "/C=CN/ST=Guangdong/L=Guangzhou/O=karmada/OU=System/CN=karmada-etcd-client" -key karmada-etcd-client.key -out karmada-etcd-client.csr openssl x509 -req -days 3650 \ -extfile <(printf "keyUsage=critical,Digital Signature, Key Encipherment\nextendedKeyUsage=clientAuth\nauthorityKeyIdentifier=keyid,issuer") \ -sha256 -CA ca.crt -CAkey ca.key -set_serial 01 -in karmada-etcd-client.csr -out karmada-etcd-client.crt ``` #### Create karmada certificate create `karmada-apiserver ` certificate. >Notice: > >If you need to access the `karmada apiserver` through the public `IP/DNS` or external `IP/DNS`, the certificate needs to be added to the `IP/DNS`. ```bash openssl genrsa -out karmada-apiserver.key 2048 openssl req -new -nodes -sha256 -subj "/C=CN/ST=Guangdong/L=Guangzhou/O=karmada/OU=System/CN=karmada" -key karmada-apiserver.key -out karmada-apiserver.csr openssl x509 -req -days 3650 \ -extfile <(printf "keyUsage=critical,Digital Signature, Key Encipherment\nextendedKeyUsage=serverAuth,clientAuth\nauthorityKeyIdentifier=keyid,issuer\nsubjectAltName=DNS:kubernetes,DNS:kubernetes.default,DNS:kubernetes.default.svc,DNS:kubernetes.default.svc.cluster,DNS:kubernetes.default.svc.cluster.local,IP:172.31.209.245,IP:172.31.209.246,IP:172.31.209.247,IP:10.254.0.1,IP:47.242.88.82") \ -sha256 -CA ca.crt -CAkey ca.key -set_serial 01 -in karmada-apiserver.csr -out karmada-apiserver.crt ``` create `karmada admin ` certificate. ```bash openssl genrsa -out admin.key 2048 openssl req -new -nodes -sha256 -subj "/C=CN/ST=Guangdong/L=Guangzhou/O=system:masters/OU=System/CN=admin" -key admin.key -out admin.csr openssl x509 -req -days 3650 \ -extfile <(printf "keyUsage=critical,Digital Signature, Key Encipherment\nextendedKeyUsage=clientAuth\nauthorityKeyIdentifier=keyid,issuer") \ -sha256 -CA ca.crt -CAkey ca.key -set_serial 01 -in admin.csr -out admin.crt ``` create `kube-controller-manager ` certificate. ```bash openssl genrsa -out kube-controller-manager.key 2048 openssl req -new -nodes -sha256 -subj "/C=CN/ST=Guangdong/L=Guangzhou/O=karmada/OU=System/CN=system:kube-controller-manager" -key kube-controller-manager.key -out kube-controller-manager.csr openssl x509 -req -days 3650 \ -extfile <(printf "keyUsage=critical,Digital Signature, Key Encipherment\nextendedKeyUsage=clientAuth\nauthorityKeyIdentifier=keyid,issuer") \ -sha256 -CA ca.crt -CAkey ca.key -set_serial 01 -in kube-controller-manager.csr -out kube-controller-manager.crt ``` create `karmada components` certificate. ```bash openssl genrsa -out karmada.key 2048 openssl req -new -nodes -sha256 -subj "/C=CN/ST=Guangdong/L=Guangzhou/O=karmada/OU=System/CN=system:karmada" -key karmada.key -out karmada.csr openssl x509 -req -days 3650 \ -extfile <(printf "keyUsage=critical,Digital Signature, Key Encipherment\nextendedKeyUsage=serverAuth,clientAuth\nauthorityKeyIdentifier=keyid,issuer\nsubjectAltName=DNS:karmada-01,DNS:karmada-02,DNS:karmada-03,DNS:localhost,IP:172.0.0.1,IP:172.31.209.245,IP:172.31.209.246,IP:172.31.209.247,IP:10.254.0.1,IP:47.242.88.82") \ -sha256 -CA ca.crt -CAkey ca.key -set_serial 01 -in karmada.csr -out karmada.crt ``` create `front-proxy-client` certificate. ```bash openssl genrsa -out front-proxy-client.key 2048 openssl req -new -nodes -sha256 -subj "/C=CN/ST=Guangdong/L=Guangzhou/O=karmada/OU=System/CN=front-proxy-client" -key front-proxy-client.key -out front-proxy-client.csr openssl x509 -req -days 3650 \ -extfile <(printf "keyUsage=critical,Digital Signature, Key Encipherment\nextendedKeyUsage=serverAuth,clientAuth\nauthorityKeyIdentifier=keyid,issuer") \ -sha256 -CA ca.crt -CAkey ca.key -set_serial 01 -in front-proxy-client.csr -out front-proxy-client.crt ``` create `karmada-apiserver` SA key ```bash openssl genrsa -out sa.key 2048 openssl rsa -in sa.key -pubout -out sa.pub ``` #### Check the certificate You can view the configuration of the certificate, take `etcd-server `as an example. ```bash openssl x509 -noout -text -in etcd-server.crt ``` #### Create the karmada configuration directory copy the certificate to the `/etc/karmada/pki` directory. ```bash mkdir -p /etc/karmada/pki cp karmada.key tls.key cp karmada.crt tls.crt cp *.key *.crt sa.pub /etc/karmada/pki ``` ## Create the karmada kubeconfig files and etcd encrypted file Execute operations at `karmada-01`. Define the karmada apiserver address. `172.31.209.245:5443` is the address of the `nginx` proxy `karmada-apiserver` ,we'll set it up later. ```bash export KARMADA_APISERVER="https://172.31.209.245:5443" cd /etc/karmada/ ``` #### Create kubectl kubeconfig file which is kept at $HOME/.kube/config by default ```bas kubectl config set-cluster karmada \ --certificate-authority=/etc/karmada/pki/ca.crt \ --embed-certs=true \ --server=${KARMADA_APISERVER} kubectl config set-credentials admin \ --client-certificate=/etc/karmada/pki/admin.crt \ --embed-certs=true \ --client-key=/etc/karmada/pki/admin.key kubectl config set-context karmada \ --cluster=karmada \ --user=admin kubectl config use-context karmada ``` #### Create kube-controller-manager kubeconfig file ```bash kubectl config set-cluster karmada \ --certificate-authority=/etc/karmada/pki/ca.crt \ --embed-certs=true \ --server=${KARMADA_APISERVER} \ --kubeconfig=kube-controller-manager.kubeconfig kubectl config set-credentials system:kube-controller-manager \ --client-certificate=/etc/karmada/pki/kube-controller-manager.crt \ --client-key=/etc/karmada/pki/kube-controller-manager.key \ --embed-certs=true \ --kubeconfig=kube-controller-manager.kubeconfig kubectl config set-context system:kube-controller-manager \ --cluster=karmada \ --user=system:kube-controller-manager \ --kubeconfig=kube-controller-manager.kubeconfig kubectl config use-context system:kube-controller-manager --kubeconfig=kube-controller-manager.kubeconfig ``` #### Create karmada kubeconfig file The components of karmada connect to the karmada apiserver through this file. ```bash kubectl config set-cluster karmada \ --certificate-authority=/etc/karmada/pki/ca.crt \ --embed-certs=true \ --server=${KARMADA_APISERVER} \ --kubeconfig=karmada.kubeconfig kubectl config set-credentials system:karmada \ --client-certificate=/etc/karmada/pki/karmada.crt \ --client-key=/etc/karmada/pki/karmada.key \ --embed-certs=true \ --kubeconfig=karmada.kubeconfig kubectl config set-context system:karmada\ --cluster=karmada \ --user=system:karmada \ --kubeconfig=karmada.kubeconfig kubectl config use-context system:karmada --kubeconfig=karmada.kubeconfig ``` #### Create etcd encrypted file ```bash export ENCRYPTION_KEY=$(head -c 32 /dev/urandom | base64) cat > /etc/karmada/encryption-config.yaml <The parameters that `karmada-02` `karmada-03` need to change are: > >--name > >--initial-advertise-peer-urls > >--listen-peer-urls > >--listen-client-urls > >--advertise-client-urls #### Start etcd cluster 3 servers have to execute. create etcd storage directory ```bash mkdir /var/lib/etcd/ chmod 700 /var/lib/etcd ``` start etcd ```bash systemctl daemon-reload systemctl enable etcd systemctl start etcd systemctl status etcd ``` #### Check etcd cluster status ```bash etcdctl --cacert=/etc/karmada/pki/ca.crt \ --cert=/etc/karmada/pki/etcd-server.crt \ --key=/etc/karmada/pki/etcd-server.key \ --endpoints 172.31.209.245:2379,172.31.209.246:2379,172.31.209.247:2379 endpoint status --write-out="table" +---------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+ | ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS | +---------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+ | 172.31.209.245:2379 | 689151f8cbf4ee95 | 3.5.1 | 20 kB | false | false | 2 | 9 | 9 | | | 172.31.209.246:2379 | 5db4dfb6ecc14de7 | 3.5.1 | 20 kB | true | false | 2 | 9 | 9 | | | 172.31.209.247:2379 | 7e59eef3c816aa57 | 3.5.1 | 20 kB | false | false | 2 | 9 | 9 | | +---------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+ ``` ## Install Karmada APIServer #### configure nginx Execute operations at `karmada-01`. configure load balancing for `karmada apiserver` ```bash cat > /usr/local/karmada-nginx/conf/nginx.conf <