From eeee80d058a01acf85b19611eac673658eb6aa70 Mon Sep 17 00:00:00 2001 From: Cassandra Coyle Date: Mon, 10 Feb 2025 10:42:12 -0600 Subject: [PATCH] add to k8s prod section about setting scheduler high priority to avoid disruption ofjobs in flight Signed-off-by: Cassandra Coyle --- .../operations/hosting/kubernetes/kubernetes-production.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/daprdocs/content/en/operations/hosting/kubernetes/kubernetes-production.md b/daprdocs/content/en/operations/hosting/kubernetes/kubernetes-production.md index 1151137ef..fa64b4386 100644 --- a/daprdocs/content/en/operations/hosting/kubernetes/kubernetes-production.md +++ b/daprdocs/content/en/operations/hosting/kubernetes/kubernetes-production.md @@ -120,13 +120,15 @@ In some scenarios, nodes may have memory and/or cpu pressure and the Dapr contro for eviction. To prevent this, you can set a critical priority class name for the Dapr control plane pods. This ensures that the Dapr control plane pods are not evicted unless all other pods with lower priority are evicted. +It's particularly important to protect the Dapr control plane components from eviction, especially the Scheduler service. When Schedulers are rescheduled or restarted, it can be highly disruptive to inflight jobs, potentially causing them to fire duplicate times. To prevent such disruptions, you should ensure the Dapr control plane components have a higher priority class than your application workloads. + Learn more about [Protecting Mission-Critical Pods](https://kubernetes.io/blog/2023/01/12/protect-mission-critical-pods-priorityclass/). There are two built-in critical priority classes in Kubernetes: - `system-cluster-critical` - `system-node-critical` (highest priority) -It's recommended to set the `priorityClassName` to `system-cluster-critical` for the Dapr control plane pods. +It's recommended to set the `priorityClassName` to `system-cluster-critical` for the Dapr control plane pods. If you have your own custom priority classes for your applications, ensure they have a lower priority value than the one assigned to the Dapr control plane to maintain system stability and prevent disruption of core Dapr services. For a new Dapr control plane deployment, the `system-cluster-critical` priority class mode can be set via the helm value `global.priorityClassName`. @@ -155,7 +157,6 @@ spec: values: [system-cluster-critical] ``` - ## Deploy Dapr with Helm [Visit the full guide on deploying Dapr with Helm]({{< ref "kubernetes-deploy.md#install-with-helm-advanced" >}}).