dapr
/
docs
mirror of https://github.com/dapr/docs.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Merge pull request #1131 from trondhindenes/helm-updates 

Helm updates
This commit is contained in:
Aaron Crawfis 2021-02-01 08:09:48 -08:00 committed by GitHub
parent f70c9645be f3d2b5b6b6
commit 8252f29792
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
3 changed files with 134 additions and 263 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

29
daprdocs/content/en/getting-started/install-dapr-kubernetes.md
View File

@ -97,7 +97,7 @@ dapr uninstall --kubernetes
## Install with Helm (advanced)
You can install Dapr to Kubernetes cluster using a Helm 3 chart.
You can install Dapr on Kubernetes using a Helm 3 chart.
{{% alert title="Note" color="primary" %}}
@ -107,31 +107,34 @@ The latest Dapr helm chart no longer supports Helm v2. Please migrate from helm
### Add and install Dapr helm chart
1. Make sure [Helm 3](https://github.com/helm/helm/releases) is installed on your machine
2. Add Helm repo and update
```bash
helm repo add dapr https://dapr.github.io/helm-charts/
helm repo update
# See which chart versions are available
helm search repo dapr --devel --versions
```
3. Create `dapr-system` namespace on your kubernetes cluster
3. Install the Dapr chart on your cluster in the `dapr-system` namespace.
```bash
kubectl create namespace dapr-system
helm upgrade --install dapr dapr/dapr \
--version=1.0.0-rc.3 \
--namespace dapr-system \
--create-namespace \
--wait
```
4. Install the Dapr chart on your cluster in the `dapr-system` namespace.
```bash
helm install dapr dapr/dapr --namespace dapr-system --version 1.0.0-rc.3
```
See [Guidelines for production ready deployments on Kubernetes]({{ ref kubernetes-production.md }}) for more information on installing and upgrading Dapr using Helm
### Verify installation
Once the chart installation is complete verify the dapr-operator, dapr-placement, dapr-sidecar-injector and dapr-sentry pods are running in the `dapr-system` namespace:
Once the chart installation is complete, verify that the dapr-operator, dapr-placement, dapr-sidecar-injector and dapr-sentry pods are running in the `dapr-system` namespace:
```bash
kubectl get pods -n dapr-system -w
kubectl get pods --namespace dapr-system
```
```
@ -146,12 +149,14 @@ dapr-sentry-9435776c7f-8f7yd 1/1 Running 0 40s
### Uninstall Dapr on Kubernetes
```bash
helm uninstall dapr -n dapr-system
helm uninstall dapr --namespace dapr-system
```
### More information
See [this page](https://github.com/dapr/dapr/blob/master/charts/dapr/README.md) for details on Dapr helm charts.
- Read [this guide]({{< ref kubernetes-production.md >}}) for recommended Helm chart values for production setups
- See [this page](https://github.com/dapr/dapr/blob/master/charts/dapr/README.md) for details on Dapr helm charts.
## Next steps

246
daprdocs/content/en/operations/hosting/kubernetes/kubernetes-production.md
View File

@ -8,28 +8,20 @@ description: "Recommendations and practices for deploying Dapr to a Kubernetes c
## Cluster capacity requirements
For a production ready Kubernetes cluster deployment, it is recommended you run a cluster of 3 worker nodes to support a highly-available setup of the control plane.
The Dapr control plane pods are designed to be lightweight and require the following resources in a production-ready setup:
For a production ready Kubernetes cluster deployment, it is recommended you run a cluster of at least 3 worker nodes to support a highly-available control plane installation.
Use the following resource settings might serve as a starting point. Requirements will vary depending on cluster size and other factors, so individual testing is needed to find the right values for your environment:
*Note: For more info on CPU and Memory resource units and their meaning, see [this](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#resource-units-in-kubernetes) link*
| Deployment | CPU | Memory
|-------------|-----|-------
| Operator | Limit: 1, Request: 100m | Limit: 200Mi, Request: 20Mi
| Sidecar Injector | Limit: 1, Request: 100m | Limit: 200Mi, Request: 20Mi
| Sentry | Limit: 1, Request: 100m | Limit: 200Mi, Request: 20Mi
| Placement | Limit: 1, Request: 250m | Limit: 500Mi, Request: 100Mi
| Operator | Limit: 1, Request: 100m | Limit: 200Mi, Request: 100Mi
| Sidecar Injector | Limit: 1, Request: 100m | Limit: 200Mi, Request: 30Mi
| Sentry | Limit: 1, Request: 100m | Limit: 200Mi, Request: 30Mi
| Placement | Limit: 1, Request: 250m | Limit: 150Mi, Request: 75Mi
| Dashboard | Limit: 200m, Request: 50m | Limit: 200Mi, Request: 20Mi
To change the resource assignments for the Dapr sidecar, see the annotations [here]({{< ref "kubernetes-annotations.md" >}}).
The specific annotations related to resource constraints are:
* `dapr.io/sidecar-cpu-limit`
* `dapr.io/sidecar-memory-limit`
* `dapr.io/sidecar-cpu-request`
* `dapr.io/sidecar-memory-request`
For more details on configuring resource in Kubernetes see [Assign Memory Resources to Containers and Pods](https://kubernetes.io/docs/tasks/configure-pod-container/assign-memory-resource/) and [Assign CPU Resources to Containers and Pods](https://kubernetes.io/docs/tasks/configure-pod-container/assign-cpu-resource/)
When installing Dapr using Helm, no default limit/request values are set. Each component has a `resources` option (for example, `dapr_dashboard.resources`), which you can use to tune the Dapr control plane to fit your environment. The [Helm chart readme](https://github.com/dapr/dapr/blob/master/charts/dapr/README) has detailed information and examples. For local/dev installations, you might simply want to skip configuring the `resources` options.
### Optional components
@ -39,30 +31,70 @@ The following Dapr control plane deployments are optional:
* Sentry - Needed for mTLS for service to service invocation
* Dashboard - Needed for operational view of the cluster
## Sidecar resource requirements
## Sidecar resource settings
The Dapr sidecar requires the following resources in a production-ready setup:
To set the resource assignments for the Dapr sidecar, see the annotations [here]({{< ref "kubernetes-annotations.md" >}}).
The specific annotations related to resource constraints are:
* `dapr.io/sidecar-cpu-limit`
* `dapr.io/sidecar-memory-limit`
* `dapr.io/sidecar-cpu-request`
* `dapr.io/sidecar-memory-request`
If not set, the dapr sidecar will run without resource settings, which may lead to issues. For a production-ready setup it is strongly recommended to configure these settings.
For more details on configuring resource in Kubernetes see [Assign Memory Resources to Containers and Pods](https://kubernetes.io/docs/tasks/configure-pod-container/assign-memory-resource/) and [Assign CPU Resources to Containers and Pods](https://kubernetes.io/docs/tasks/configure-pod-container/assign-cpu-resource/)
Example settings for the dapr sidecar in a production-ready setup:
| CPU | Memory |
|-----|--------|
| Limit: 4, Request: 100m | Limit: 4000Mi, Request: 250Mi
| Limit: 300m, Request: 100m | Limit: 1000Mi, Request: 250Mi
*Note: Since Dapr is intended to do much of the I/O heavy lifting for your app, it's expected that the resources given to Dapr enable you to drastically reduce the resource allocations for the application*
The CPU and memory limits above account for the fact that Dapr is intended to do a lot of high performant I/O bound operations. Based on your app needs, you can increase or decrease those limits accordingly.
The CPU and memory limits above account for the fact that Dapr is intended to a high number of I/O bound operations. It is strongly recommended that you use a tool monitoring tool to baseline the sidecar (and app) containers and tune these settings based on those baselines.
## Deploying Dapr with Helm
When deploying to a production cluster, it's recommended to use Helm. The Dapr CLI installation into a Kubernetes cluster is for a development and test only setup.
When deploying to a production cluster, it's recommended to use Helm. Although the Dapr CLI can install Dapr onto a Kubernetes cluster, it is intended for use in dev/test scenarios.
You can find information [here]({{< ref "install-dapr-selfhost.md#using-helm-advanced" >}}) on how to deploy Dapr using Helm.
When deploying Dapr in a production-ready configuration, it's recommended to deploy with a highly available configuration of the control plane:
When deploying Dapr in a production-ready configuration, it's recommended to deploy with a highly available configuration of the control plane. It is recommended to create a values file instead of specifying parameters on the command-line. This file should be checked in to source control so that you can track changes made to it.
For a full list of all available options you can set in the values file (or by using the `--set` command-line option), see https://github.com/dapr/dapr/blob/master/charts/dapr/README.md.
Instead of using either `helm install` or `helm upgrade` as shown below, you can also run `helm upgrade --install` - this will dynamically determine whether to install or upgrade.
```bash
helm install dapr dapr/dapr --version=<Dapr chart version> --namespace dapr-system --set global.ha.enabled=true
# add/update the helm repo
helm repo add dapr https://dapr.github.io/helm-charts/
helm repo update
# See which chart versions are available
helm search repo dapr --devel --versions
# create a values file to store variables
touch values.yml
cat << EOF >> values.yml
global.ha.enabled: true
EOF
# run install/upgrade
helm install dapr dapr/dapr \
--version=<Dapr chart version> \
--namespace dapr-system \
--create-namespace \
--values values.yml \
--wait
# verify the installation
kubectl get pods --namespace dapr-system
```
This command will run 3 replicas of each control plane pod in the dapr-system namespace.
This command will run 3 replicas of each control plane service in the dapr-system namespace.
*Note: The Dapr Helm chart automatically deploys with affinity for nodes with the label `kubernetes.io/os=linux`. You can deploy the Dapr control plane to Windows nodes, but most users should not need to. For more information see [Deploying to a Hybrid Linux/Windows K8s Cluster]({{< ref "kubernetes-hybrid-clusters.md" >}})*
@ -78,110 +110,11 @@ Dapr supports zero downtime upgrades. The upgrade path includes the following st
To upgrade the Dapr CLI, [download the latest version](https://github.com/dapr/cli/releases) of the CLI. After you downloaded the binary, it's recommended you put the CLI binary in your path.
### Updating the control plane
### Upgrading the control plane
#### Saving the current certificates
See [Steps to upgrade Dapr on a Kubernetes cluster]({{< ref kubernetes-upgrade.md >}})
When upgrading to a new version of Dapr, it is recommended you carry over the root and issuer certificates instead of re-generating them, which might cause a downtime for applications that make use of service invocation or actors.
#### Exporting certs with the Dapr CLI
To get your current certs with the Dapr CLI, run the following command:
```
dapr mtls export -o ./certs
```
This will save any existing root cert, issuer cert and issuer key in the output dir of your choice.
### Exporting certs manually
To get the current root and issuer certificates, run the following command:
```
kubectl get secret dapr-trust-bundle -o yaml -n dapr-system
apiVersion: v1
data:
ca.crt: <ROOT-CERTIFICATE-VALUE>
issuer.crt: <ISSUER-CERTIFICATE-VALUE>
issuer.key: <ISSUER-KEY-VALUE>
kind: Secret
```
Copy the contents of `ca.crt`, `issuer.crt` and `issuer.key` and base64 decode them. Save these certificates as files.
You should have the following files containing the base64 decoded text from the secret saved on your disk:
1. ca.crt
2. issuer.crt
3. issuer.key
#### Updating the control plane pods
> Note: To upgrade Dapr from 0.11.x to 1.0.0 version, please refer to [this section](#upgrade-from-dapr-011x-to-100).
Next, you need to find a Helm chart version that installs the new desired version of Dapr and perform a `helm upgrade` operation.
First, update the Helm Chart repos:
```bash
helm repo update
```
List all charts in the Dapr repo:
```bash
helm search repo dapr --devel
NAME CHART VERSION APP VERSION DESCRIPTION
dapr/dapr 1.0.0-rc.1 1.0.0-rc.1 A Helm chart for Dapr on Kubernetes
```
The APP VERSION column tells us which Dapr runtime version is installed by the chart. Now, use the following command to upgrade Dapr to your desired runtime version providing a path to the certificate files you saved before:
> Remove `--set global.ha.enabled=true` if current Dapr installation has not been deployed in HA mode.
```bash
helm upgrade dapr dapr/dapr \
--version <Dapr chart version> \
--namespace dapr-system \
--reset-values \
--set-file dapr_sentry.tls.root.certPEM=certs/ca.crt \
--set-file dapr_sentry.tls.issuer.certPEM=certs/issuer.crt \
--set-file dapr_sentry.tls.issuer.keyPEM=certs/issuer.key \
--set global.ha.enabled=true
```
Kubernetes now performs a rolling update. Wait until all the new pods appear as running:
```bash
kubectl get po -n dapr-system -w
NAME READY STATUS RESTARTS AGE
dapr-dashboard-86b94bb768-w4wmj 1/1 Running 0 39s
dapr-operator-67d7d7bb6c-qqkk7 1/1 Running 0 39s
dapr-placement-server-0 1/1 Running 0 39s
dapr-sentry-647759cd46-nwzkw 1/1 Running 0 39s
dapr-sidecar-injector-74648c9dcb-px2m5 1/1 Running 0 39s
```
You can verify the health and version of the control plane using the Dapr CLI:
```bash
dapr status -k
NAME NAMESPACE HEALTHY STATUS REPLICAS VERSION AGE CREATED
dapr-sidecar-injector dapr-system True Running 1 1.0.0-rc.1 1m 2020-11-16 14:42.19
dapr-sentry dapr-system True Running 1 1.0.0-rc.1 1m 2020-11-16 14:42.19
dapr-dashboard dapr-system True Running 1 0.3.0 1m 2020-11-16 14:42.19
dapr-operator dapr-system True Running 1 1.0.0-rc.1 1m 2020-11-16 14:42.19
dapr-placement-server dapr-system True Running 1 1.0.0-rc.1 1m 2020-11-16 14:42.19
```
*Note: If new fields have been added to the target Helm Chart being upgraded to, the `helm upgrade` command will fail. If that happens, you need to find which new fields have been added in the new chart and add them as parameters to the upgrade command, for example: `--set <field-name>=<value>`.*
#### Updating the data plane (sidecars)
### Updating the data plane (sidecars)
The last step is to update pods that are running Dapr to pick up the new version of the Dapr runtime.
To do that, simply issue a rollout restart command for any deployment that has the `dapr.io/enabled` annotation:
@ -199,76 +132,21 @@ APP ID APP PORT AGE CREATED
nodeapp 3000 16h 2020-07-29 17:16.22
```
#### Upgrade from Dapr 0.11.x to 1.0.0
Run the below commands first to migrate from 0.11.x placement service safely:
```sh
kubectl annotate deployment dapr-placement "helm.sh/resource-policy"=keep -n dapr-system
kubectl annotate svc dapr-placement "helm.sh/resource-policy"=keep -n dapr-system
```
Then [export certs manually](#exporting-certs-manually).
```sh
dapr mtls export -o ./certs
```
Upgrade Dapr using the below commands; this example upgrades Dapr from 0.11.x to 1.0.0-rc.1 with HA mode.
```sh
helm repo update
helm upgrade dapr dapr/dapr --version 1.0.0-rc.1 --namespace dapr-system --reset-values --set-file dapr_sentry.tls.root.certPEM=./certs/ca.crt --set-file dapr_sentry.tls.issuer.certPEM=./certs/issuer.crt --set-file dapr_sentry.tls.issuer.keyPEM=./certs/issuer.key --set global.ha.enabled=true --wait
```
Once Dapr is installed completely, ensure that 0.11.x dapr-placement is still running and **wait until all pods are running**
```sh
kubectl get pods -n dapr-system -w
NAME READY STATUS RESTARTS AGE
dapr-dashboard-69f5c5c867-mqhg4 1/1 Running 0 42s
dapr-operator-5cdd6b7f9c-9sl7g 1/1 Running 0 41s
dapr-operator-5cdd6b7f9c-jkzjs 1/1 Running 0 29s
dapr-operator-5cdd6b7f9c-qzp8n 1/1 Running 0 34s
dapr-placement-5dcb574777-nlq4t 1/1 Running 0 76s ---- 0.11.x placement
dapr-placement-server-0 1/1 Running 0 41s
dapr-placement-server-1 1/1 Running 0 41s
dapr-placement-server-2 1/1 Running 0 41s
dapr-sentry-84565c747b-7bh8h 1/1 Running 0 35s
dapr-sentry-84565c747b-fdlls 1/1 Running 0 41s
dapr-sentry-84565c747b-ldnsf 1/1 Running 0 29s
dapr-sidecar-injector-68f868668f-6xnbt 1/1 Running 0 41s
dapr-sidecar-injector-68f868668f-j7jcq 1/1 Running 0 29s
dapr-sidecar-injector-68f868668f-ltxq4 1/1 Running 0 36s
```
Update pods that are running Dapr to pick up the new version of the Dapr runtime.
```sh
kubectl rollout restart deploy/<Application deployment name>
```
Once the deployment is completed, delete 0.11.x dapr-placement service by following commands:
```sh
kubectl delete deployment dapr-placement -n dapr-system
kubectl delete svc dapr-placement -n dapr-system
```
## Recommended security configuration
Properly configured, Dapr not only be secured with regards to it's control plane and sidecars communication, but can also make your application more secure with a number of built-in features.
When properly configured, Dapr ensures secure communication. It can also make your application more secure with a number of built-in features.
It is recommended that a production-ready deployment includes the following settings:
1. Mutual Authentication (mTLS) should be enabled. Note that Dapr has mTLS on by default. For details on how to bring your own certificates, see [here]({{< ref "mtls.md#bringing-your-own-certificates" >}})
2. Dapr API authentication is enabled (this is the between your application and the Dapr sidecar). To secure the Dapr API from unauthorized access, it is recommended to enable Dapr's token based auth. See [here]({{< ref "api-token.md" >}}) for details
2. Dapr API authentication is enabled (this is the communication between your application and the Dapr sidecar). To secure the Dapr API from unauthorized access, it is recommended to enable Dapr's token based auth. See [here]({{< ref "api-token.md" >}}) for details
3. All component YAMLs should have secret data configured in a secret store and not hard-coded in the YAML file. See [here]({{< ref "component-secrets.md" >}}) on how to use secrets with Dapr components
4. The Dapr control plane is installed on a separate namespace such as `dapr-system`, and never into the `default` namespace
4. The Dapr control plane is installed on a dedicated namespace such as `dapr-system`.
Dapr also supports scoping components for certain applications. This is not a required practice, and can be enabled according to your Sec-Ops needs. See [here]({{< ref "component-scopes.md" >}}) for more info.
Dapr also supports scoping components for certain applications. This is not a required practice, and can be enabled according to your security needs. See [here]({{< ref "component-scopes.md" >}}) for more info.
## Tracing and metrics configuration

122
daprdocs/content/en/operations/hosting/kubernetes/kubernetes-upgrade.md
View File

@ -8,10 +8,55 @@ description: "Follow these steps to upgrade Dapr on Kubernetes and ensure a smoo
## Prerequisites
- Latest [Dapr CLI]({{< ref install-dapr-cli.md >}})
- [Helm 3](https://github.com/helm/helm/releases)
## Upgrade existing cluster running 0.11.x
## Upgrade existing cluster to 1.0.0 / 1.0.0-rc.3
From version 1.0.0-rc.3 onwards, upgrading Dapr using Helm is no longer a disruptive action since existing certificate values will automatically be re-used.
1. Upgrade Dapr from 1.0.0-rc.1 (or newer) to 1.0.0-rc.3 (or newer):
```bash
helm repo update
```
```bash
helm upgrade dapr dapr/dapr --version 1.0.0-rc.3 --namespace dapr-system --set global.ha.enabled=true --wait
```
*If you're using a values file, remember to add the `--values` option when running the upgrade command.*
2. Ensure all pods are running:
```bash
kubectl get pods -n dapr-system -w
NAME READY STATUS RESTARTS AGE
dapr-dashboard-69f5c5c867-mqhg4 1/1 Running 0 42s
dapr-operator-5cdd6b7f9c-9sl7g 1/1 Running 0 41s
dapr-operator-5cdd6b7f9c-jkzjs 1/1 Running 0 29s
dapr-operator-5cdd6b7f9c-qzp8n 1/1 Running 0 34s
dapr-placement-server-0 1/1 Running 0 41s
dapr-placement-server-1 1/1 Running 0 41s
dapr-placement-server-2 1/1 Running 0 41s
dapr-sentry-84565c747b-7bh8h 1/1 Running 0 35s
dapr-sentry-84565c747b-fdlls 1/1 Running 0 41s
dapr-sentry-84565c747b-ldnsf 1/1 Running 0 29s
dapr-sidecar-injector-68f868668f-6xnbt 1/1 Running 0 41s
dapr-sidecar-injector-68f868668f-j7jcq 1/1 Running 0 29s
dapr-sidecar-injector-68f868668f-ltxq4 1/1 Running 0 36s
```
3. Restart your application deployments to update the Dapr runtime:
```bash
kubectl rollout restart deploy/<DEPLOYMENT-NAME>
```
4. All done!
## Upgrade existing cluster to 1.0.0-rc.2
If you have an older Dapr version installed and want to upgrade to a version lower than 1.0.0-rc.3 (for example, upgrade from 0.11.1 to 1.0.0-rc.2), follow these steps:
1. Run these two commands to prevent `helm upgrade` from uninstalling `0.11.x` placement service:
@ -22,22 +67,22 @@ description: "Follow these steps to upgrade Dapr on Kubernetes and ensure a smoo
kubectl annotate svc dapr-placement helm.sh/resource-policy=keep -n dapr-system
```
1. Export certificates:
2. Export certificates:
```bash
dapr mtls export -o ./certs
```
1. Upgrade Dapr to 1.0.0-rc.3:
3. Upgrade Dapr to 1.0.0-rc.2:
```bash
helm repo update
```
```bash
helm upgrade dapr dapr/dapr --version 1.0.0-rc.3 --namespace dapr-system --reset-values --set-file dapr_sentry.tls.root.certPEM=./certs/ca.crt --set-file dapr_sentry.tls.issuer.certPEM=./certs/issuer.crt --set-file dapr_sentry.tls.issuer.keyPEM=./certs/issuer.key --set global.ha.enabled=true --wait
helm upgrade dapr dapr/dapr --version 1.0.0-rc.2 --namespace dapr-system --reset-values --set-file dapr_sentry.tls.root.certPEM=./certs/ca.crt --set-file dapr_sentry.tls.issuer.certPEM=./certs/issuer.crt --set-file dapr_sentry.tls.issuer.keyPEM=./certs/issuer.key --set global.ha.enabled=true --wait
```
1. Upgrade CRDs:
4. Upgrade CRDs:
```bash
kubectl replace -f https://raw.githubusercontent.com/dapr/dapr/21636a9237f2dcecd9c80f329e99b512e8377739/charts/dapr/crds/configuration.yaml
@ -46,7 +91,7 @@ description: "Follow these steps to upgrade Dapr on Kubernetes and ensure a smoo
kubectl replace -f https://raw.githubusercontent.com/dapr/dapr/21636a9237f2dcecd9c80f329e99b512e8377739/charts/dapr/crds/components.yaml
```
1. Ensure 0.11.x dapr-placement service is still running and wait until all pods are running:
5. Ensure 0.11.x dapr-placement service is still running and wait until all pods are running:
```bash
kubectl get pods -n dapr-system -w
@ -68,13 +113,13 @@ description: "Follow these steps to upgrade Dapr on Kubernetes and ensure a smoo
dapr-sidecar-injector-68f868668f-ltxq4 1/1 Running 0 36s
```
1. Restart your application deployments to update the Dapr runtime.
6. Restart your application deployments to update the Dapr runtime.
```bash
kubectl rollout restart deploy/<DEPLOYMENT-NAME>
```
1. Once the deployment is completed, delete the 0.11.x dapr-placement service:
7. Once the deployment is completed, delete the 0.11.x dapr-placement service:
```bash
kubectl delete deployment dapr-placement -n dapr-system
@ -84,64 +129,7 @@ description: "Follow these steps to upgrade Dapr on Kubernetes and ensure a smoo
kubectl delete svc dapr-placement -n dapr-system
```
1. All done!
## Upgrade existing cluster running 1.0.0-rc.1
1. Export certs:
```bash
dapr mtls export -o ./certs
```
1. Upgrade Dapr to 1.0.0-3:
```bash
helm repo update
```
```bash
helm upgrade dapr dapr/dapr --version 1.0.0-rc.3 --namespace dapr-system --reset-values --set-file dapr_sentry.tls.root.certPEM=./certs/ca.crt --set-file dapr_sentry.tls.issuer.certPEM=./certs/issuer.crt --set-file dapr_sentry.tls.issuer.keyPEM=./certs/issuer.key --set global.ha.enabled=true --wait
```
1. Upgrade CRDs:
```bash
kubectl replace -f https://raw.githubusercontent.com/dapr/dapr/21636a9237f2dcecd9c80f329e99b512e8377739/charts/dapr/crds/configuration.yaml
```
```bash
kubectl replace -f https://raw.githubusercontent.com/dapr/dapr/21636a9237f2dcecd9c80f329e99b512e8377739/charts/dapr/crds/components.yaml
```
1. Ensure all pods are running:
```bash
kubectl get pods -n dapr-system -w
NAME READY STATUS RESTARTS AGE
dapr-dashboard-69f5c5c867-mqhg4 1/1 Running 0 42s
dapr-operator-5cdd6b7f9c-9sl7g 1/1 Running 0 41s
dapr-operator-5cdd6b7f9c-jkzjs 1/1 Running 0 29s
dapr-operator-5cdd6b7f9c-qzp8n 1/1 Running 0 34s
dapr-placement-server-0 1/1 Running 0 41s
dapr-placement-server-1 1/1 Running 0 41s
dapr-placement-server-2 1/1 Running 0 41s
dapr-sentry-84565c747b-7bh8h 1/1 Running 0 35s
dapr-sentry-84565c747b-fdlls 1/1 Running 0 41s
dapr-sentry-84565c747b-ldnsf 1/1 Running 0 29s
dapr-sidecar-injector-68f868668f-6xnbt 1/1 Running 0 41s
dapr-sidecar-injector-68f868668f-j7jcq 1/1 Running 0 29s
dapr-sidecar-injector-68f868668f-ltxq4 1/1 Running 0 36s
```
1. Restart your application deployments to update the Dapr runtime:
```bash
kubectl rollout restart deploy/<DEPLOYMENT-NAME>
```
1. All done!
8. All done!
## Next steps