diff --git a/docs/installation/install-binary.md b/docs/installation/install-binary.md new file mode 100644 index 000000000..610721c41 --- /dev/null +++ b/docs/installation/install-binary.md @@ -0,0 +1,1069 @@ +Step-by-step installation of binary high-availability `karmada` cluster. + +# Installing Karmada cluster + +## Prerequisites + +#### server + +3 servers required. E.g. + +```shell ++---------------+-----------------+-----------------+ +| HostName | Host IP | Public IP | ++---------------+-----------------+-----------------+ +| karmada-01 | 172.31.209.245 | 47.242.88.82 | ++---------------+-----------------+-----------------+ +| karmada-02 | 172.31.209.246 | | ++---------------+-----------------+-----------------+ +| karmada-03 | 172.31.209.247 | | ++---------------+-----------------+-----------------+ +``` + +> Public IP is not required. It is used to download some `karmada` dependent components from the public network and connect to `karmada` ApiServer through the public network + +#### hosts parsing + +Execute operations at `karmada-01` `karmada-02` `karmada-03`. + +```bash +vi /etc/hosts +172.31.209.245 karmada-01 +172.31.209.246 karmada-02 +172.31.209.247 karmada-03 +``` + +#### environment + +`karmada-01` requires the following environment. + +**Golang**: Compile the karmada binary +**GCC**: Compile nginx (ignore if using cloud load balancing) + + + + + + + +## Compile and download binaries + +Execute operations at `karmada-01`. + +#### kubernetes binaries + +Download the `kubernetes` binary package. + +```bash +wget https://dl.k8s.io/v1.23.3/kubernetes-server-linux-amd64.tar.gz +tar -zxvf kubernetes-server-linux-amd64.tar.gz +cd /root/kubernetes/server/bin +mv kube-apiserver kube-controller-manager kubectl /usr/local/sbin/ +``` + +#### etcd binaries + +Download the `etcd` binary package. + +```bash +wget https://github.com/etcd-io/etcd/releases/download/v3.5.1/etcd-v3.5.1-linux-amd64.tar.gz +tar -zxvf etcd-v3.5.1-linux-amd64.tar.gz +cd etcd-v3.5.1-linux-amd64/ +cp etcdctl etcd /usr/local/sbin/ +``` + +#### karmada binaries + +Compile the `karmada` binary from source. + +```bash +git clone https://github.com/karmada-io/karmada +cd karmada +make karmada-aggregated-apiserver +make karmada-controller-manager +make karmada-scheduler +make karmada-webhook +mv karmada-aggregated-apiserver karmada-controller-manager karmada-scheduler karmada-webhook /usr/local/sbin/ +``` + +#### nginx binaries + +Compile the `nginx` binary from source. + +```bash +wget http://nginx.org/download/nginx-1.21.6.tar.gz +tar -zxvf nginx-1.21.6.tar.gz +cd nginx-1.21.6 +./configure --with-stream --without-http --prefix=/usr/local/karmada-nginx --without-http_uwsgi_module --without-http_scgi_module --without-http_fastcgi_module +make && make install +mv /usr/local/karmada-nginx/sbin/nginx /usr/local/karmada-nginx/sbin/karmada-nginx +``` + +#### Distribute binaries + +Upload the binary file to the `karmada-02` `karmada-03 ` server + +```bash +scp /usr/local/sbin/* karmada-02:/usr/local/sbin/ +scp /usr/local/sbin/* karmada-03:/usr/local/sbin/ +``` + + + +## Generate certificate + +Generated using the `openssl` command. Note yes `DNS` and `IP` when generating the certificate. + +Execute operations at `karmada-01`. + +#### create a temporary directory for certificates + +```bash +mkdir certs +cd certs +``` + +#### Create root certificate + +valid for 10 years + +```bash +openssl genrsa -out ca.key 2048 +openssl req -x509 -new -nodes -key ca.key -subj "/C=CN/ST=Guangdong/L=Guangzhou/O=karmada/OU=System/CN=karmada" -days 3650 -out ca.crt +``` + +#### Create etcd certificate + +create `etcd server ` certificate + +```bash +openssl genrsa -out etcd-server.key 2048 +openssl req -new -nodes -sha256 -subj "/C=CN/ST=Guangdong/L=Guangzhou/O=karmada/OU=System/CN=karmada-etcd" -key etcd-server.key -out etcd-server.csr +openssl x509 -req -days 3650 \ + -extfile <(printf "keyUsage=critical,Digital Signature, Key Encipherment\nextendedKeyUsage=serverAuth,clientAuth\nauthorityKeyIdentifier=keyid,issuer\nsubjectAltName=IP:172.31.209.245,IP:172.31.209.246,IP:172.31.209.247,IP:127.0.0.1,DNS:localhost") \ + -sha256 -CA ca.crt -CAkey ca.key -set_serial 01 -in etcd-server.csr -out etcd-server.crt +``` + +create `etcd peer ` certificate + +```bash +openssl genrsa -out etcd-peer.key 2048 +openssl req -new -nodes -sha256 -subj "/C=CN/ST=Guangdong/L=Guangzhou/O=karmada/OU=System/CN=karmada-etcd-peer" -key etcd-peer.key -out etcd-peer.csr +openssl x509 -req -days 3650 \ + -extfile <(printf "keyUsage=critical,Digital Signature, Key Encipherment\nextendedKeyUsage=serverAuth,clientAuth\nauthorityKeyIdentifier=keyid,issuer\nsubjectAltName=IP:172.31.209.245,IP:172.31.209.246,IP:172.31.209.247,IP:127.0.0.1,DNS:localhost") \ + -sha256 -CA ca.crt -CAkey ca.key -set_serial 01 -in etcd-peer.csr -out etcd-peer.crt +``` + +create `etcd client ` certificate + +```bash +openssl genrsa -out karmada-etcd-client.key 2048 +openssl req -new -nodes -sha256 -subj "/C=CN/ST=Guangdong/L=Guangzhou/O=karmada/OU=System/CN=karmada-etcd-client" -key karmada-etcd-client.key -out karmada-etcd-client.csr +openssl x509 -req -days 3650 \ + -extfile <(printf "keyUsage=critical,Digital Signature, Key Encipherment\nextendedKeyUsage=clientAuth\nauthorityKeyIdentifier=keyid,issuer") \ + -sha256 -CA ca.crt -CAkey ca.key -set_serial 01 -in karmada-etcd-client.csr -out karmada-etcd-client.crt +``` + +#### Create karmada certificate + +create `karmada-apiserver ` certificate. + +>Notice: +> +>If you need to access the `karmada apiserver` through the public `IP/DNS` or external `IP/DNS`, the certificate needs to be added to the `IP/DNS`. + +```bash +openssl genrsa -out karmada-apiserver.key 2048 +openssl req -new -nodes -sha256 -subj "/C=CN/ST=Guangdong/L=Guangzhou/O=karmada/OU=System/CN=karmada" -key karmada-apiserver.key -out karmada-apiserver.csr +openssl x509 -req -days 3650 \ + -extfile <(printf "keyUsage=critical,Digital Signature, Key Encipherment\nextendedKeyUsage=serverAuth,clientAuth\nauthorityKeyIdentifier=keyid,issuer\nsubjectAltName=DNS:kubernetes,DNS:kubernetes.default,DNS:kubernetes.default.svc,DNS:kubernetes.default.svc.cluster,DNS:kubernetes.default.svc.cluster.local,IP:172.31.209.245,IP:172.31.209.246,IP:172.31.209.247,IP:10.254.0.1,IP:47.242.88.82") \ + -sha256 -CA ca.crt -CAkey ca.key -set_serial 01 -in karmada-apiserver.csr -out karmada-apiserver.crt +``` + +create `karmada admin ` certificate. + +```bash +openssl genrsa -out admin.key 2048 +openssl req -new -nodes -sha256 -subj "/C=CN/ST=Guangdong/L=Guangzhou/O=system:masters/OU=System/CN=admin" -key admin.key -out admin.csr +openssl x509 -req -days 3650 \ + -extfile <(printf "keyUsage=critical,Digital Signature, Key Encipherment\nextendedKeyUsage=clientAuth\nauthorityKeyIdentifier=keyid,issuer") \ + -sha256 -CA ca.crt -CAkey ca.key -set_serial 01 -in admin.csr -out admin.crt +``` + +create `kube-controller-manager ` certificate. + +```bash +openssl genrsa -out kube-controller-manager.key 2048 +openssl req -new -nodes -sha256 -subj "/C=CN/ST=Guangdong/L=Guangzhou/O=karmada/OU=System/CN=system:kube-controller-manager" -key kube-controller-manager.key -out kube-controller-manager.csr +openssl x509 -req -days 3650 \ + -extfile <(printf "keyUsage=critical,Digital Signature, Key Encipherment\nextendedKeyUsage=clientAuth\nauthorityKeyIdentifier=keyid,issuer") \ + -sha256 -CA ca.crt -CAkey ca.key -set_serial 01 -in kube-controller-manager.csr -out kube-controller-manager.crt +``` + +create `karmada components` certificate. + +```bash +openssl genrsa -out karmada.key 2048 +openssl req -new -nodes -sha256 -subj "/C=CN/ST=Guangdong/L=Guangzhou/O=karmada/OU=System/CN=system:karmada" -key karmada.key -out karmada.csr +openssl x509 -req -days 3650 \ + -extfile <(printf "keyUsage=critical,Digital Signature, Key Encipherment\nextendedKeyUsage=serverAuth,clientAuth\nauthorityKeyIdentifier=keyid,issuer\nsubjectAltName=DNS:karmada-01,DNS:karmada-02,DNS:karmada-03,DNS:localhost,IP:172.0.0.1,IP:172.31.209.245,IP:172.31.209.246,IP:172.31.209.247,IP:10.254.0.1,IP:47.242.88.82") \ + -sha256 -CA ca.crt -CAkey ca.key -set_serial 01 -in karmada.csr -out karmada.crt +``` + +create `front-proxy-client` certificate. + +```bash +openssl genrsa -out front-proxy-client.key 2048 +openssl req -new -nodes -sha256 -subj "/C=CN/ST=Guangdong/L=Guangzhou/O=karmada/OU=System/CN=front-proxy-client" -key front-proxy-client.key -out front-proxy-client.csr +openssl x509 -req -days 3650 \ + -extfile <(printf "keyUsage=critical,Digital Signature, Key Encipherment\nextendedKeyUsage=serverAuth,clientAuth\nauthorityKeyIdentifier=keyid,issuer") \ + -sha256 -CA ca.crt -CAkey ca.key -set_serial 01 -in front-proxy-client.csr -out front-proxy-client.crt +``` + +create `karmada-apiserver` SA key + +```bash +openssl genrsa -out sa.key 2048 +openssl rsa -in sa.key -pubout -out sa.pub +``` + +#### Check the certificate + +You can view the configuration of the certificate, take `etcd-server `as an example. + +```bash +openssl x509 -noout -text -in etcd-server.crt +``` + +#### Create the karmada configuration directory + +copy the certificate to the `/etc/karmada/pki` directory. + +```bash +mkdir -p /etc/karmada/pki +cp karmada.key tls.key +cp karmada.crt tls.crt +cp *.key *.crt sa.pub /etc/karmada/pki +``` + + + +## Create the karmada kubeconfig files and etcd encrypted file + +Execute operations at `karmada-01`. + +Define the karmada apiserver address. `172.31.209.245:5443` is the address of the `nginx` proxy `karmada-apiserver` ,we'll set it up later. + +```bash +export KARMADA_APISERVER="https://172.31.209.245:5443" +cd /etc/karmada/ +``` + +#### Create kubectl kubeconfig file + +which is kept at $HOME/.kube/config by default + +```bas +kubectl config set-cluster karmada \ + --certificate-authority=/etc/karmada/pki/ca.crt \ + --embed-certs=true \ + --server=${KARMADA_APISERVER} + +kubectl config set-credentials admin \ + --client-certificate=/etc/karmada/pki/admin.crt \ + --embed-certs=true \ + --client-key=/etc/karmada/pki/admin.key + +kubectl config set-context karmada \ + --cluster=karmada \ + --user=admin + +kubectl config use-context karmada +``` + +#### Create kube-controller-manager kubeconfig file + +```bash +kubectl config set-cluster karmada \ + --certificate-authority=/etc/karmada/pki/ca.crt \ + --embed-certs=true \ + --server=${KARMADA_APISERVER} \ + --kubeconfig=kube-controller-manager.kubeconfig + +kubectl config set-credentials system:kube-controller-manager \ + --client-certificate=/etc/karmada/pki/kube-controller-manager.crt \ + --client-key=/etc/karmada/pki/kube-controller-manager.key \ + --embed-certs=true \ + --kubeconfig=kube-controller-manager.kubeconfig + +kubectl config set-context system:kube-controller-manager \ + --cluster=karmada \ + --user=system:kube-controller-manager \ + --kubeconfig=kube-controller-manager.kubeconfig + +kubectl config use-context system:kube-controller-manager --kubeconfig=kube-controller-manager.kubeconfig +``` + +#### Create karmada kubeconfig file + +The components of karmada connect to the karmada apiserver through this file. + +```bash +kubectl config set-cluster karmada \ + --certificate-authority=/etc/karmada/pki/ca.crt \ + --embed-certs=true \ + --server=${KARMADA_APISERVER} \ + --kubeconfig=karmada.kubeconfig + +kubectl config set-credentials system:karmada \ + --client-certificate=/etc/karmada/pki/karmada.crt \ + --client-key=/etc/karmada/pki/karmada.key \ + --embed-certs=true \ + --kubeconfig=karmada.kubeconfig + +kubectl config set-context system:karmada\ + --cluster=karmada \ + --user=system:karmada \ + --kubeconfig=karmada.kubeconfig + +kubectl config use-context system:karmada --kubeconfig=karmada.kubeconfig +``` + +#### Create etcd encrypted file + +```bash +export ENCRYPTION_KEY=$(head -c 32 /dev/urandom | base64) +cat > /etc/karmada/encryption-config.yaml <The parameters that `karmada-02` `karmada-03` need to change are: +> +>--name +> +>--initial-advertise-peer-urls +> +>--listen-peer-urls +> +>--listen-client-urls +> +>--advertise-client-urls + + + +#### Start etcd cluster + +3 servers have to execute. + +create etcd storage directory + +```bash +mkdir /var/lib/etcd/ +chmod 700 /var/lib/etcd +``` + +start etcd + +```bash +systemctl daemon-reload +systemctl enable etcd +systemctl start etcd +systemctl status etcd +``` + +#### Check etcd cluster status + +```bash +etcdctl --cacert=/etc/karmada/pki/ca.crt \ + --cert=/etc/karmada/pki/etcd-server.crt \ + --key=/etc/karmada/pki/etcd-server.key \ + --endpoints 172.31.209.245:2379,172.31.209.246:2379,172.31.209.247:2379 endpoint status --write-out="table" + ++---------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+ +| ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS | ++---------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+ +| 172.31.209.245:2379 | 689151f8cbf4ee95 | 3.5.1 | 20 kB | false | false | 2 | 9 | 9 | | +| 172.31.209.246:2379 | 5db4dfb6ecc14de7 | 3.5.1 | 20 kB | true | false | 2 | 9 | 9 | | +| 172.31.209.247:2379 | 7e59eef3c816aa57 | 3.5.1 | 20 kB | false | false | 2 | 9 | 9 | | ++---------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+ +``` + + + +## Install Karmada APIServer + +#### configure nginx + +Execute operations at `karmada-01`. + +configure load balancing for `karmada apiserver` + +```bash +cat > /usr/local/karmada-nginx/conf/nginx.conf <