karmada/charts/karmada

Karmada

Karmada (Kubernetes Armada) is a Kubernetes management system that enables you to run your cloud-native applications across multiple Kubernetes clusters and clouds, with no changes to your applications. By speaking Kubernetes-native APIs and providing advanced scheduling capabilities, Karmada enables truly open, multi-cloud Kubernetes.

Karmada aims to provide turnkey automation for multi-cluster application management in multi-cloud and hybrid cloud scenarios, with key features such as centralized multi-cloud management, high availability, failure recovery, and traffic scheduling.

TL;DR

Switch to the root directory of the repo.

helm install karmada -n karmada-system --create-namespace --dependency-update ./charts/karmada

Prerequisites

• Kubernetes 1.16+
• helm v3+

Installing the Chart

There are two installation methods, local and remote, to install the chart with the release name karmada in namespace karmada-system.

Local installation

Switch to the root directory of the repo.

helm install karmada -n karmada-system --create-namespace --dependency-update ./charts/karmada

Remote installation

First, add the Karmada chart repo to your local repository.

$ helm repo add karmada-charts https://raw.githubusercontent.com/karmada-io/karmada/master/charts
$ helm repo list
NAME            URL
karmada-charts   https://raw.githubusercontent.com/karmada-io/karmada/master/charts

With the repo added, available charts and versions can be viewed.

helm search repo karmada

Install the chart and specify the version to install with the --version argument. Replace <x.x.x> with your desired version.

helm --namespace karmada-system upgrade -i karmada karmada-charts/karmada --version=<x.x.x> --create-namespace
Release "karmada" does not exist. Installing it now.
NAME: karmada
LAST DEPLOYED: Mon May 30 07:19:36 2022
NAMESPACE: karmada-system
STATUS: deployed
REVISION: 1
TEST SUITE: None

Export kubeconfig

Get kubeconfig from the cluster:

kubectl get secret -n karmada-system karmada-kubeconfig -o jsonpath={.data.kubeconfig} | base64 -d

Tip: List all releases using helm list

Install descheduler component

For better scheduling effect, you can choose to install the karmada-descheduler component in the karmada controller plane. We can install it together in host mode:

helm upgrade --install karmada -n karmada-system --create-namespace --dependency-update \
     --cleanup-on-fail ./charts/karmada \
     --set components={"descheduler"}

And then, check the pod status of karmada:

kubectl get po -n karmada-system
NAME                                              READY   STATUS    RESTARTS      AGE
etcd-0                                            1/1     Running   0             2m12s
karmada-aggregated-apiserver-7769446f86-hc2jk     1/1     Running   2 (65s ago)   2m12s
karmada-apiserver-55bdd6b9c5-8s999                1/1     Running   0             2m12s
karmada-controller-manager-764cfd8d6c-lm67w       1/1     Running   2 (67s ago)   2m12s
karmada-descheduler-59749dc655-d4wsv              1/1     Running   0             2m12s
karmada-descheduler-59749dc655-sf9b7              1/1     Running   0             2m12s
karmada-kube-controller-manager-7b5cc7897-64lrp   1/1     Running   2 (42s ago)   2m12s
karmada-scheduler-66566879d5-xqqlc                1/1     Running   0             2m12s
karmada-webhook-749f5f75df-4l5h4                  1/1     Running   2 (67s ago)   2m12s

Uninstalling the Chart

To uninstall/delete the karmada helm release in namespace karmada-system:

helm uninstall karmada -n karmada-system

The command removes all the Kubernetes components associated with the chart and deletes the release.

Note: There are some RBAC resources that are used by the preJob that can not be deleted by the uninstall command above. You might have to clean them manually with tools like kubectl. You can clean them by commands:

kubectl delete sa/karmada-hook-job -nkarmada-system
kubectl delete clusterRole/karmada-hook-job
kubectl delete clusterRoleBinding/karmada-hook-job
kubectl delete ns karmada-system

Example

1. Install agent

Edited values.yaml

installMode: "agent"
agent:
  clusterName: "member"
  clusterEndpoint: "https://X.X.X.X:6443"
  ## kubeconfig of the karmada
  kubeconfig:
    caCrt: |
      -----BEGIN CERTIFICATE-----
      XXXXXXXXXXXXXXXXXXXXXXXXXXX
      -----END CERTIFICATE-----      
    crt: |
      -----BEGIN CERTIFICATE-----
      XXXXXXXXXXXXXXXXXXXXXXXXXXX
      -----END CERTIFICATE-----      
    key: |
      -----BEGIN RSA PRIVATE KEY-----
      XXXXXXXXXXXXXXXXXXXXXXXXXXX
      -----END RSA PRIVATE KEY-----      
    server: "https://apiserver.karmada"

Execute command (switch to the root directory of the repo, and sets the current-context in a kubeconfig file)

kubectl config use-context member
helm install karmada-agent -n karmada-system --create-namespace ./charts/karmada

2. Install component

Edited values.yaml for karmada-scheduler-estimator

installMode: "component"
components: [
  "schedulerEstimator"
]
schedulerEstimator:
  memberClusters:
  - clusterName: "member"
    replicaCount: 1
    ## kubeconfig of the member cluster
    kubeconfig:
      caCrt: |
        -----BEGIN CERTIFICATE-----
        XXXXXXXXXXXXXXXXXXXXXXXXXXX
        -----END CERTIFICATE-----        
      crt: |
        -----BEGIN CERTIFICATE-----
        XXXXXXXXXXXXXXXXXXXXXXXXXXX
        -----END CERTIFICATE-----        
      key: |
        -----BEGIN RSA PRIVATE KEY-----
        XXXXXXXXXXXXXXXXXXXXXXXXXXX
        -----END RSA PRIVATE KEY-----        
      server: "https://apiserver.member"

Execute command (switch to the root directory of the repo, and sets the current-context in a kubeconfig file)

kubectl config use-context host
helm install karmada-scheduler-estimator -n karmada-system ./charts/karmada

Configuration

Global parameters

Name	Description	Value
global.imageRegistry	Global Docker image registry	""
global.imagePullSecrets	Global Docker registry secret names as an array	[]

Common parameters

Name	Description	Value
installMode	InstallMode "host", "agent" and "component" are provided, "host" means install karmada in the control-cluster, "agent" means install agent client in the member cluster, "component" means install selected components in the control-cluster	"host"
clusterDomain	Default cluster domain for karmada	"cluster.local"
components	Selected components list, selectable values: "schedulerEstimator"	[]
kubectl.image.repository	Image of the kubectl	"bitnami/kubectl"
kubectl.image.tag	Image tag of the kubectl	"latest"
kubectl.image.pullPolicy	Image pull policy of the kubectl	"Always"
kubectl.image.pullSecrets	Image pull secret of the kubectl	[]
cfssl.image.repository	Image of the cfssl	"cfssl/cfssl"
cfssl.image.tag	Image tag of the cfssl	"latest"
cfssl.image.pullPolicy	Image pull policy of the cfssl	"Always"
cfssl.image.pullSecrets	Image pull secret of the cfssl	[]
certs.mode	Mode "auto" and "custom" are provided, "auto" means auto generate certificate, "custom" means use user certificate	"auto"
certs.auto.expiry	Expiry of the certificate	"43800h"
certs.auto.hosts	Hosts of the certificate	["kubernetes.default.svc","*.etcd.{{ .Release.Namespace }}.svc.{{ .Values.clusterDomain }}","*.{{ .Release.Namespace }}.svc.{{ .Values.clusterDomain }}","*.{{ .Release.Namespace }}.svc","localhost","127.0.0.1"]
certs.custom.caCrt	CA CRT of the certificate	""
certs.custom.crt	CRT of the certificate	""
certs.custom.key	KEY of the certificate	""
certs.custom.frontProxyCaCrt	CA CRT of the front proxy certificate	""
certs.custom.frontProxyCrt	CRT of the front proxy certificate	""
certs.custom.frontProxyKey	KEY of the front proxy certificate	""
etcd.affinity	Affinity of the etcd pods	{}
etcd.labels	Labels of etcd statefulset	{"app": "etcd"}
etcd.podLabels	Labels of etcd pods	{}
etcd.mode	Mode "external" and "internal" are provided, "external" means use external ectd, "internal" means install a etcd in the cluster	"internal"
etcd.external.servers	Servers of etcd	""
etcd.external.registryPrefix	Use to registry prefix of etcd	"/registry/karmada"
etcd.external.certs.caCrt	CA CRT of the etcd certificate	""
etcd.external.certs.crt	CRT of the etcd certificate	""
etcd.external.certs.key	KEY of the etcd certificate	""
etcd.internal.replicaCount	Target replicas of the etcd	1
etcd.internal.image.repository	Image of the etcd	"registry.k8s.io/etcd"
etcd.internal.image.pullPolicy	Image pull policy of the etcd	"IfNotPresent"
etcd.internal.image.tag	Image tag of the etcd	"3.5.16-0"
etcd.internal.storageType	StorageType of the etcd, accepts "hostPath", "pvc"	"hostPath"
etcd.internal.pvc.storageClass	StorageClass of the etcd, takes effect whenetcd.internal.storageType is "pvc"	""
etcd.internal.pvc.size	Storage size of the etcd, takes effect whenetcd.internal.storageType is "pvc"	""
etcd.internal.nodeSelector	Node selector of the karmada-apiserver	{}
etcd.internal.resources	Resource quota of the etcd	{}
etcd.internal.tolerations	Tolerations of the internal etcd	[]
agent.clusterName	Name of the member cluster	""
agent.clusterEndpoint	Server endpoint of the member cluster	""
agent.kubeconfig.caCrt	CA CRT of the karmada certificate	""
agent.kubeconfig.crt	CRT of the karmada certificate	""
agent.kubeconfig.key	KEY of the karmada certificate	""
agent.kubeconfig.server	API-server of the karmada	""
agent.labels	Labels of the agent deployment	{"app": "karmada-agent"}
agent.replicaCount	Target replicas of the agent	1
agent.podLabels	Labels of the agent pods	{}
agent.podAnnotations	Annotations of the agent pods	{}
agent.image.pullSecrets	Image pull secret of the agent	[]
agent.image.repository	Image of the agent	"docker.io/karmada/karmada-agent"
agent.image.tag	Image tag of the agent	"latest"
agent.image.pullPolicy	Image pull policy of the agent	"Always"
agent.resources	Resource quota of the agent	{}
agent.nodeSelector	Node selector of the agent	{}
agent.affinity	Affinity of the agent	{}
agent.tolerations	Tolerations of the agent	[]
agent.strategy	Strategy of the agent	{"type": "RollingUpdate", "rollingUpdate": {"maxUnavailable": "0", "maxSurge": "50%"} }
scheduler.labels	Labels of the scheduler deployment	{"app": "karmada-scheduler"}
scheduler.replicaCount	Target replicas of the scheduler	1
scheduler.podLabels	Labels of the scheduler pods	{}
scheduler.podAnnotations	Annotations of the scheduler pods	{}
scheduler.image.pullSecrets	Image pull secret of the scheduler	[]
scheduler.image.repository	Image of the scheduler	"docker.io/karmada/karmada-scheduler"
scheduler.image.tag	Image tag of the scheduler	"latest"
scheduler.image.pullPolicy	Image pull policy of the scheduler	"Always"
scheduler.resources	Resource quota of the scheduler	{}
scheduler.nodeSelector	Node selector of the scheduler	{}
scheduler.affinity	Affinity of the scheduler	{}
scheduler.tolerations	Tolerations of the scheduler	[]
scheduler.strategy	Strategy of the scheduler	{"type": "RollingUpdate", "rollingUpdate": {"maxUnavailable": "0", "maxSurge": "50%"} }
scheduler.enableSchedulerEstimator	Enable calling cluster scheduler estimator for adjusting replicas	false
webhook.labels	Labels of the webhook deployment	{"app": "karmada-webhook"}
webhook.replicaCount	Target replicas of the webhook	1
webhook.podLabels	Labels of the webhook pods	{}
webhook.podAnnotations	Annotations of the webhook pods	{}
webhook.image.pullSecrets	Image pull secret of the webhook	[]
webhook.image.repository	Image of the webhook	"docker.io/karmada/karmada-webhook"
webhook.image.tag	Image tag of the webhook	"latest"
webhook.image.pullPolicy	Image pull policy of the webhook	"Always"
webhook.resources	Resource quota of the webhook	{}
webhook.nodeSelector	Node selector of the webhook	{}
webhook.affinity	Affinity of the webhook	{}
webhook.tolerations	Tolerations of the webhook	[]
webhook.strategy	Strategy of the webhook	{"type": "RollingUpdate", "rollingUpdate": {"maxUnavailable": "0", "maxSurge": "50%"} }
controllerManager.labels	Labels of the karmada-controller-manager deployment	{"app": "karmada-controller-manager"}
controllerManager.replicaCount	Target replicas of the karmada-controller-manager	1
controllerManager.podLabels	Labels of the karmada-controller-manager pods	{}
controllerManager.podAnnotations	Annotations of the karmada-controller-manager pods	{}
controllerManager.image.pullSecrets	Image pull secret of the karmada-controller-manager	[]
controllerManager.image.repository	Image of the karmada-controller-manager	"docker.io/karmada/karmada-controller-manager"
controllerManager.image.tag	Image tag of the karmada-controller-manager	"latest"
controllerManager.image.pullPolicy	Image pull policy of the karmada-controller-manager	"Always"
controllerManager.resources	Resource quota of the karmada-controller-manager	{}
controllerManager.nodeSelector	Node selector of the karmada-controller-manager	{}
controllerManager.affinity	Affinity of the karmada-controller-manager	{}
controllerManager.tolerations	Tolerations of the karmada-controller-manager	[]
controllerManager.strategy	Strategy of the karmada-controller-manager	{"type": "RollingUpdate", "rollingUpdate": {"maxUnavailable": "0", "maxSurge": "50%"} }
controllerManager.featureGates	FeatureGates of the karmada-controller-manager	{"PropagateDeps": "false"}
controllerManager.controllers	Controllers of the karmada-controller-manager	""
controllerManager.extraCommandArgs	extra command args of the karmada-controller-manager	{}
apiServer.labels	Labels of the karmada-apiserver deployment	{"app": "karmada-apiserver"}
apiServer.serviceAnnotations	Annotations of the karmada-apiserver service	{}
apiServer.replicaCount	Target replicas of the karmada-apiserver	1
apiServer.podLabels	Labels of the karmada-apiserver pods	{}
apiServer.podAnnotations	Annotations of the karmada-apiserver pods	{}
apiServer.image.pullSecrets	Image pull secret of the karmada-apiserver	[]
apiServer.image.repository	Image of the karmada-apiserver	"registry.k8s.io/kube-apiserver"
apiServer.image.tag	Image tag of the karmada-apiserver	"v1.31.3"
apiServer.image.pullPolicy	Image pull policy of the karmada-apiserver	"IfNotPresent"
apiServer.resources	Resource quota of the karmada-apiserver	{}
apiServer.hostNetwork	Deploy karmada-apiserver with hostNetwork. If there are multiple karmadas in one cluster, you'd better set it to "false"	"false"
apiServer.nodeSelector	Node selector of the karmada-apiserver	{}
apiServer.affinity	Affinity of the karmada-apiserver	{}
apiServer.tolerations	Tolerations of the karmada-apiserver	[]
apiServer.serviceType	Service type of apiserver, accepts "ClusterIP", "NodePort", "LoadBalancer"	"ClusterIP"
apiServer.nodePort	Node port for apiserver, takes effect whenapiServer.serviceType is "NodePort". If no port is specified, the nodePort will be automatically assigned.	0
apiServer.strategy	Strategy of the karmada-apiserver	{"type": "RollingUpdate", "rollingUpdate": {"maxUnavailable": "0", "maxSurge": "50%"} }
aggregatedApiServer.labels	Labels of the karmada-aggregated-apiserver deployment	{"app": "karmada-aggregated-apiserver"}
aggregatedApiServer.replicaCount	Target replicas of the karmada-aggregated-apiserver	1
aggregatedApiServer.podLabels	Labels of the karmada-aggregated-apiserver pods	{}
aggregatedApiServer.podAnnotations	Annotations of the karmada-aggregated-apiserver pods	{}
aggregatedApiServer.image.pullSecrets	Image pull secret of the karmada-aggregated-apiserver	[]
aggregatedApiServer.image.repository	Image of the karmada-aggregated-apiserver	"docker.io/karmada/karmada-aggregated-apiserver"
aggregatedApiServer.image.tag	Image tag of the karmada-aggregated-apiserver	"latest"
aggregatedApiServer.image.pullPolicy	Image pull policy of the karmada-aggregated-apiserver	"Always"
aggregatedApiServer.resources	Resource quota of the karmada-aggregated-apiserver	{requests: {cpu: 100m}}
aggregatedApiServer.nodeSelector	Node selector of the karmada-aggregated-apiserver	{}
aggregatedApiServer.affinity	Affinity of the karmada-aggregated-apiserver	{}
aggregatedApiServer.tolerations	Tolerations of the karmada-aggregated-apiserver	[]
aggregatedApiServer.strategy	Strategy of the karmada-aggregated-apiserver	{"type": "RollingUpdate", "rollingUpdate": {"maxUnavailable": "0", "maxSurge": "50%"} }
kubeControllerManager.labels	Labels of the kube-controller-manager deployment	{"app": "kube-controller-manager"}
kubeControllerManager.replicaCount	Target replicas of the kube-controller-manager	1
kubeControllerManager.podLabels	Labels of the kube-controller-manager pods	{}
kubeControllerManager.podAnnotations	Annotations of the kube-controller-manager pods	{}
kubeControllerManager.image.pullSecrets	Image pull secret of the kube-controller-manager	[]
kubeControllerManager.image.repository	Image of the kube-controller-manager	"registry.k8s.io/kube-controller-manager"
kubeControllerManager.image.tag	Image tag of the kube-controller-manager	"v1.31.3"
kubeControllerManager.image.pullPolicy	Image pull policy of the kube-controller-manager	"IfNotPresent"
kubeControllerManager.resources	Resource quota of the kube-controller-manager	{}
kubeControllerManager.nodeSelector	Node selector of the kube-controller-manager	{}
kubeControllerManager.affinity	Affinity of the kube-controller-manager	{}
kubeControllerManager.tolerations	Tolerations of the kube-controller-manager	[]
kubeControllerManager.strategy	Strategy of the kube-controller-manager	{"type": "RollingUpdate", "rollingUpdate": {"maxUnavailable": "0", "maxSurge": "50%"} }
schedulerEstimator.clusterName	Name of the member cluster	""
schedulerEstimator.kubeconfig.caCrt	CA CRT of the certificate	""
schedulerEstimator.kubeconfig.crt	CRT of the certificate	""
schedulerEstimator.kubeconfig.key	KEY of the certificate	""
schedulerEstimator.kubeconfig.server	API-server of the member cluster	""
schedulerEstimator.labels	Labels of the scheduler-estimator deployment	{}
schedulerEstimator.replicaCount	Target replicas of the scheduler-estimator	1
schedulerEstimator.podLabels	Labels of the scheduler-estimator pods	{}
schedulerEstimator.podAnnotations	Annotations of the scheduler-estimator pods	{}
schedulerEstimator.image.pullSecrets	Image pull secret of the scheduler-estimator	[]
schedulerEstimator.image.repository	Image of the scheduler-estimator	"docker.io/karmada/karmada-scheduler-estimator"
schedulerEstimator.image.tag	Image tag of the scheduler-estimator	"latest"
schedulerEstimator.image.pullPolicy	Image pull policy of the scheduler-estimator	"Always"
schedulerEstimator.resources	Resource quota of the scheduler-estimator	{}
schedulerEstimator.nodeSelector	Node selector of the scheduler-estimator	{}
schedulerEstimator.affinity	Affinity of the scheduler-estimator	{}
schedulerEstimator.tolerations	Tolerations of the scheduler-estimator	[]
schedulerEstimator.featureGates	FeatureGates of the scheduler-estimator	{"FeatureGateName": "false"}
search.strategy	Strategy of the scheduler-estimator	{"type": "RollingUpdate", "rollingUpdate": {"maxUnavailable": "0", "maxSurge": "50%"} }
descheduler.labels	Labels of the descheduler deployment	karmada-descheduler
descheduler.replicaCount	Target replicas of the descheduler	2
descheduler.podAnnotations	Annotations of the descheduler pods	{}
descheduler.podLabels	Labels of the descheduler pods	{}
descheduler.image.pullSecrets	Image pull secret of the descheduler	[]
descheduler.image.repository	Image of the descheduler	docker.io/karmada/karmada-descheduler
descheduler.image.pullPolicy	Image pull policy of the descheduler	Always
descheduler.image.tag	Overrides the image tag whose default is the latest	latest
descheduler.resources	Resource quota of the descheduler	{}
descheduler.nodeSelector	Node selector of the descheduler	{}
descheduler.affinity	Affinity of the descheduler	{}
descheduler.tolerations	Tolerations of the descheduler	{}
descheduler.strategy	Strategy of the descheduler	{"type": "RollingUpdate", "rollingUpdate": {"maxUnavailable": "0", "maxSurge": "50%"} }
descheduler.kubeconfig	Kubeconfig of the descheduler	karmada-kubeconfig
search.labels	Labels of the search deployment	{"app": "karmada-search", "apiserver": "true"}
search.replicaCount	Target replicas of the search	2
search.podAnnotations	Annotations of the search pods	{}
search.podLabels	Labels of the search pods	{}
search.image.pullSecrets	Image pull secret of the search	[]
search.image.repository	Image of the search	docker.io/karmada/karmada-search
search.image.pullPolicy	Image pull policy of the search	Always
search.image.tag	Overrides the image tag whose default is the latest	latest
search.resources	Resource quota of the search	{}
search.nodeSelector	Node selector of the search	{}
search.affinity	Affinity of the search	{}
search.tolerations	Tolerations of the search	{}
search.strategy	Strategy of the search	{"type": "RollingUpdate", "rollingUpdate": {"maxUnavailable": "0", "maxSurge": "50%"} }
search.certs	Certs of the search	karmada-cert
search.kubeconfig	Kubeconfig of the search	karmada-kubeconfig
preInstallJob.tolerations	Tolerations of pre-install-job	[]
preInstallJob.nodeSelector	NodeSelector of pre-install-job	{}
postInstallJob.tolerations	Tolerations of post-install-job	[]
postInstallJob.nodeSelector	NodeSelector of post-install-job	{}
postDeleteJob.tolerations	Tolerations of post-delete-job	[]
postDeleteJob.nodeSelector	NodeSelector of post-delete-job	{}