mirror of https://github.com/dapr/docs.git
Add docs for sidecar injector watchdog (#2601)
* Added docs for sidecar injector watchdog Fixes #2581 Co-authored-by: Mark Fussell <markfussell@gmail.com> Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com> * Added YouTube video Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com> Co-authored-by: Mark Fussell <markfussell@gmail.com>
This commit is contained in:
Alessandro (Ale) Segala
GitHub
parent
09b7e18dc0
commit
f6cb49523c
|
|
@ -9,4 +9,43 @@ When running Dapr in [Kubernetes mode]({{< ref kubernetes >}}), a pod running th
|
|||
|
||||
## Running the operator service
|
||||
|
||||
The operator service is deployed as part of `dapr init -k`, or via the Dapr Helm charts. For more information on running Dapr on Kubernetes, visit the [Kubernetes hosting page]({{< ref kubernetes >}}).
|
||||
The operator service is deployed as part of `dapr init -k`, or via the Dapr Helm charts. For more information on running Dapr on Kubernetes, visit the [Kubernetes hosting page]({{< ref kubernetes >}}).
|
||||
|
||||
## Additional configuration options
|
||||
|
||||
The operator service includes additional configuration options.
|
||||
|
||||
### Injector watchdog
|
||||
|
||||
The operator service includes an _injector watchdog_ feature which periodically polls all pods running in your Kubernetes cluster and confirms that the Dapr sidecar is injected in those which have the `dapr.io/enabled=true` annotation. It is primarily meant to address situations where the [Injector service]({{< ref sidecar-injector >}}) did not successfully inject the sidecar (the `daprd` container) into pods.
|
||||
|
||||
|
||||
The injector watchdog can be useful in a few situations, including:
|
||||
|
||||
- Recovering from a Kubernetes cluster completely stopped. When a cluster is completely stopped and then restarted (including in the case of a total cluster failure), pods are restarted in a random order. If your application is restarted before the Dapr control plane (specifically the Injector service) is ready, the Dapr sidecar may not be injected into your application's pods, causing your application to behave unexpectedly.
|
||||
|
||||
- Addressing potential random failures with the sidecar injector, such as transient failures within the Injector service.
|
||||
|
||||
|
||||
If the watchdog detects a pod that does not have a sidecar when it should have had one, it deletes it. Kubernetes will then re-create the pod, invoking the Dapr sidecar injector again.
|
||||
|
||||
The injector watchdog feature is **disabled by default**.
|
||||
|
||||
You can enable it by passing the `--watch-interval` flag to the `operator` command, which can take one of the following values:
|
||||
|
||||
|
||||
- `--watch-interval=0`: disables the injector watchdog (default value if the flag is omitted).
|
||||
- `--watch-interval=<interval>`: the injector watchdog is enabled and polls all pods at the given interval; the value for the interval is a string that includes the unit. For example: `--watch-interval=10s` (every 10 seconds) or `--watch-interval=2m` (every 2 minutes).
|
||||
- `--watch-interval=once`: the injector watchdog runs only once when the operator service is started.
|
||||
|
||||
If you're using Helm, you can configure the injector watchdog with the [`dapr_operator.watchInterval` option](https://github.com/dapr/dapr/blob/master/charts/dapr/README.md#dapr-operator-options), which has the same values as the command line flags.
|
||||
|
||||
|
||||
> The injector watchdog is safe to use when the operator service is running in HA (High Availability) mode with more than one replica. In this case, Kubernetes automatically elects a "leader" instance which is the only one that runs the injector watchdog service.
|
||||
|
||||
> However, when in HA mode, if you configure the injector watchdog to run "once", the watchdog polling is actually started every time an instance of the operator service is elected as leader. This means that, should the leader of the operator service crash and a new leader be elected, that would trigger the injector watchdog again.
|
||||
|
||||
Watch this video for an overview of the injector watchdog:
|
||||
|
||||
<div class="embed-responsive embed-responsive-16by9">
|
||||
<iframe width="360" height="315" src="https://www.youtube-nocookie.com/embed/ecFvpp24lpo?start=1848" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
|
||||
|
|
|
|||
|
|
@ -8,8 +8,9 @@ description: "Recommendations and practices for deploying Dapr to a Kubernetes c
|
|||
|
||||
## Cluster capacity requirements
|
||||
|
||||
For a production-ready Kubernetes cluster deployment, it is recommended you run a cluster of at least 3 worker nodes to support a highly-available control plane installation.
|
||||
Use the following resource settings as a starting point. Requirements will vary depending on cluster size and other factors, so perform individual testing to find the right values for your environment:
|
||||
For a production-ready Kubernetes cluster deployment, we recommended you run a cluster of at least 3 worker nodes to support a highly-available control plane installation.
|
||||
|
||||
Use the following resource settings as a starting point. Requirements will vary depending on cluster size and other factors, so you should perform individual testing to find the right values for your environment:
|
||||
|
||||
| Deployment | CPU | Memory
|
||||
|-------------|-----|-------
|
||||
|
|
@ -21,20 +22,23 @@ Use the following resource settings as a starting point. Requirements will vary
|
|||
|
||||
{{% alert title="Note" color="primary" %}}
|
||||
For more info, read the [concept article on CPU and Memory resource units and their meaning](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#resource-units-in-kubernetes).
|
||||
|
||||
{{% /alert %}}
|
||||
|
||||
### Helm
|
||||
|
||||
When installing Dapr using Helm, no default limit/request values are set. Each component has a `resources` option (for example, `dapr_dashboard.resources`), which you can use to tune the Dapr control plane to fit your environment. The [Helm chart readme](https://github.com/dapr/dapr/blob/master/charts/dapr/README.md) has detailed information and examples. For local/dev installations, you might simply want to skip configuring the `resources` options.
|
||||
When installing Dapr using Helm, no default limit/request values are set. Each component has a `resources` option (for example, `dapr_dashboard.resources`), which you can use to tune the Dapr control plane to fit your environment.
|
||||
|
||||
The [Helm chart readme](https://github.com/dapr/dapr/blob/master/charts/dapr/README.md) has detailed information and examples.
|
||||
|
||||
For local/dev installations, you might simply want to skip configuring the `resources` options.
|
||||
|
||||
### Optional components
|
||||
|
||||
The following Dapr control plane deployments are optional:
|
||||
|
||||
- **Placement** - Needed for Dapr Actors
|
||||
- **Sentry** - Needed for mTLS for service to service invocation
|
||||
- **Dashboard** - Needed for operational view of the cluster
|
||||
- **Placement**: needed to use Dapr Actors
|
||||
- **Sentry**: needed for mTLS for service to service invocation
|
||||
- **Dashboard**: needed to get an operational view of the cluster
|
||||
|
||||
## Sidecar resource settings
|
||||
|
||||
|
|
@ -58,14 +62,13 @@ Example settings for the Dapr sidecar in a production-ready setup:
|
|||
|
||||
{{% alert title="Note" color="primary" %}}
|
||||
Since Dapr is intended to do much of the I/O heavy lifting for your app, it's expected that the resources given to Dapr enable you to drastically reduce the resource allocations for the application.
|
||||
|
||||
{{% /alert %}}
|
||||
|
||||
The CPU and memory limits above account for the fact that Dapr is intended to a high number of I/O bound operations. It is strongly recommended that you use a monitoring tool to baseline the sidecar (and app) containers and tune these settings based on those baselines.
|
||||
The CPU and memory limits above account for the fact that Dapr is intended to support a high number of I/O bound operations. It is strongly recommended that you use a monitoring tool to get a baseline for the sidecar (and app) containers and tune these settings based on those baselines.
|
||||
|
||||
## Highly-available mode
|
||||
|
||||
When deploying Dapr in a production-ready configuration, it's recommended to deploy with a highly available (HA) configuration of the control plane, which creates 3 replicas of each control plane pod in the dapr-system namespace. This configuration allows the Dapr control plane to retain 3 running instances and survive node failures and other outages.
|
||||
When deploying Dapr in a production-ready configuration, it is recommend to deploy with a highly available (HA) configuration of the control plane, which creates 3 replicas of each control plane pod in the dapr-system namespace. This configuration allows the Dapr control plane to retain 3 running instances and survive individual node failures and other outages.
|
||||
|
||||
For a new Dapr deployment, the HA mode can be set with both the [Dapr CLI]({{< ref "kubernetes-deploy.md#install-in-highly-available-mode" >}}) and with [Helm charts]({{< ref "kubernetes-deploy.md#add-and-install-dapr-helm-chart" >}}).
|
||||
|
||||
|
|
@ -76,6 +79,7 @@ For an existing Dapr deployment, enabling the HA mode requires additional steps.
|
|||
[Visit the full guide on deploying Dapr with Helm]({{< ref "kubernetes-deploy.md#install-with-helm-advanced" >}}).
|
||||
|
||||
### Parameters file
|
||||
|
||||
Instead of specifying parameters on the command line, it's recommended to create a values file. This file should be checked into source control so that you can track its changes.
|
||||
|
||||
For a full list of all available options you can set in the values file (or by using the `--set` command-line option), see https://github.com/dapr/dapr/blob/master/charts/dapr/README.md.
|
||||
|
|
@ -119,7 +123,7 @@ The Dapr Helm chart automatically deploys with affinity for nodes with the label
|
|||
|
||||
## Upgrading Dapr with Helm
|
||||
|
||||
Dapr supports zero downtime upgrades. The upgrade path includes the following steps:
|
||||
Dapr supports zero-downtime upgrades. The upgrade path includes the following steps:
|
||||
|
||||
1. Upgrading a CLI version (optional but recommended)
|
||||
2. Updating the Dapr control plane
|
||||
|
|
@ -189,7 +193,6 @@ It is recommended that a production-ready deployment includes the following sett
|
|||
|
||||
6. Dapr also supports **scoping components for certain applications**. This is not a required practice, and can be enabled according to your security needs. See [here]({{< ref "component-scopes.md" >}}) for more info.
|
||||
|
||||
|
||||
## Tracing and metrics configuration
|
||||
|
||||
Dapr has tracing and metrics enabled by default. It is *recommended* that you set up distributed tracing and metrics for your applications and the Dapr control plane in production.
|
||||
|
|
@ -197,15 +200,26 @@ Dapr has tracing and metrics enabled by default. It is *recommended* that you se
|
|||
If you already have your own observability set-up, you can disable tracing and metrics for Dapr.
|
||||
|
||||
### Tracing
|
||||
|
||||
To configure a tracing backend for Dapr visit [this]({{< ref "setup-tracing.md" >}}) link.
|
||||
|
||||
### Metrics
|
||||
|
||||
For metrics, Dapr exposes a Prometheus endpoint listening on port 9090 which can be scraped by Prometheus.
|
||||
|
||||
To setup Prometheus, Grafana and other monitoring tools with Dapr, visit [this]({{< ref "monitoring" >}}) link.
|
||||
|
||||
## Injector watchdog
|
||||
|
||||
The Dapr Operator service includes an _injector watchdog_ which can be used to detect and remediate situations where your application's pods may be deployed without the Dapr sidecar (the `daprd` container) when they should have been. For example, it can assist with recovering the applications after a total cluster failure.
|
||||
|
||||
The injector watchdog is disabled by default when running Dapr in Kubernetes mode and it is recommended that you consider enabling it with values that are appropriate for your specific situation.
|
||||
|
||||
Refer to the documentation for the [Dapr operator]({{< ref operator >}}) service for more details on the injector watchdog and how to enable it.
|
||||
|
||||
## Best Practices
|
||||
|
||||
Watch this video for a deep dive into the best practices for running Dapr in production with Kubernetes
|
||||
|
||||
<div class="embed-responsive embed-responsive-16by9">
|
||||
<iframe width="360" height="315" src="https://www.youtube.com/embed/_U9wJqq-H1g" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
|
||||
<iframe width="360" height="315" src="https://www.youtube-nocookie.com/embed/_U9wJqq-H1g" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
|
||||
|
|
|
|||
Loading…
Reference in New Issue