mirror of https://github.com/dapr/docs.git
Update docs for low-cardinality metrics
Fixes #3987 Includes documenting the new "http.increasedCardinality" configuration option Signed-off-by: ItalyPaleAle <43508+ItalyPaleAle@users.noreply.github.com>
This commit is contained in:
ItalyPaleAle
parent
72e139d394
commit
e1d36a3698
|
|
@ -106,21 +106,27 @@ The `metrics` section under the `Configuration` spec contains the following prop
|
|||
```yml
|
||||
metrics:
|
||||
enabled: true
|
||||
rules: []
|
||||
http:
|
||||
increasedCardinality: true
|
||||
```
|
||||
|
||||
The following table lists the properties for metrics:
|
||||
|
||||
| Property | Type | Description |
|
||||
|--------------|--------|-------------|
|
||||
| `enabled` | boolean | Whether metrics should to be enabled. |
|
||||
| `rules` | boolean | Named rule to filter metrics. Each rule contains a set of `labels` to filter on and a`regex`expression to apply to the metrics path. |
|
||||
| `enabled` | boolean | When set to true, the default, enables metrics collection and the metrics endpoint. |
|
||||
| `rules` | array | Named rule to filter metrics. Each rule contains a set of `labels` to filter on and a`regex`expression to apply to the metrics path. |
|
||||
| `http.increasedCardinality` | boolean | When set to true, in the Dapr HTTP server each request path causes the creation of a new "bucket" of metrics. This can cause issues, including excessive memory consumption, when there many different requested endpoints (such as when interacting with RESTful APIs).<br>In Dapr 1.13 the default value is `true` (to preserve the behavior of Dapr <= 1.12), but will change to `false` in Dapr 1.14. |
|
||||
|
||||
To mitigate high memory usage and egress costs associated with [high cardinality metrics]({{< ref "metrics-overview.md#high-cardinality-metrics" >}}), you can set regular expressions for every metric exposed by the Dapr sidecar. For example:
|
||||
To mitigate high memory usage and egress costs associated with [high cardinality metrics]({{< ref "metrics-overview.md#high-cardinality-metrics" >}}) with the HTTP server, you should set the `metrics.http.increasedCardinality` property to `false`.
|
||||
|
||||
Using rules, you can set regular expressions for every metric exposed by the Dapr sidecar. For example:
|
||||
|
||||
```yml
|
||||
metric:
|
||||
enabled: true
|
||||
rules:
|
||||
metrics:
|
||||
enabled: true
|
||||
rules:
|
||||
- name: dapr_runtime_service_invocation_req_sent_total
|
||||
labels:
|
||||
- name: method
|
||||
|
|
|
|||
|
|
@ -3,22 +3,24 @@ type: docs
|
|||
title: "Configure metrics"
|
||||
linkTitle: "Overview"
|
||||
weight: 4000
|
||||
description: "Enable or disable Dapr metrics "
|
||||
description: "Enable or disable Dapr metrics"
|
||||
---
|
||||
|
||||
By default, each Dapr system process emits Go runtime/process metrics and has their own [Dapr metrics](https://github.com/dapr/dapr/blob/master/docs/development/dapr-metrics.md).
|
||||
|
||||
## Prometheus endpoint
|
||||
The Dapr sidecars exposes a [Prometheus](https://prometheus.io/) metrics endpoint that you can scrape to gain a greater understanding of how Dapr is behaving.
|
||||
|
||||
The Dapr sidecars exposes a [Prometheus](https://prometheus.io/)-compatible metrics endpoint that you can scrape to gain a greater understanding of how Dapr is behaving.
|
||||
|
||||
## Configuring metrics using the CLI
|
||||
|
||||
The metrics application endpoint is enabled by default. You can disable it by passing the command line argument `--enable-metrics=false`.
|
||||
|
||||
The default metrics port is `9090`. You can override this by passing the command line argument `--metrics-port` to Daprd.
|
||||
The default metrics port is `9090`. You can override this by passing the command line argument `--metrics-port` to daprd.
|
||||
|
||||
## Configuring metrics in Kubernetes
|
||||
You can also enable/disable the metrics for a specific application by setting the `dapr.io/enable-metrics: "false"` annotation on your application deployment. With the metrics exporter disabled, `daprd` does not open the metrics listening port.
|
||||
|
||||
You can also enable/disable the metrics for a specific application by setting the `dapr.io/enable-metrics: "false"` annotation on your application deployment. With the metrics exporter disabled, daprd does not open the metrics listening port.
|
||||
|
||||
The following Kubernetes deployment example shows how metrics are explicitly enabled with the port specified as "9090".
|
||||
|
||||
|
|
@ -54,10 +56,8 @@ spec:
|
|||
```
|
||||
|
||||
## Configuring metrics using application configuration
|
||||
You can also enable metrics via application configuration. To disable the metrics collection in the Dapr sidecars running in a specific namespace:
|
||||
|
||||
- Use the `metrics` spec configuration.
|
||||
- Set `enabled: false` to disable the metrics in the Dapr runtime.
|
||||
You can also enable metrics via application configuration. To disable the metrics collection in the Dapr sidecars by default, set `spec.metrics.enabled` to `false`.
|
||||
|
||||
```yaml
|
||||
apiVersion: dapr.io/v1alpha1
|
||||
|
|
@ -66,15 +66,23 @@ metadata:
|
|||
name: tracing
|
||||
namespace: default
|
||||
spec:
|
||||
tracing:
|
||||
samplingRate: "1"
|
||||
metrics:
|
||||
enabled: false
|
||||
```
|
||||
|
||||
## High cardinality metrics
|
||||
|
||||
Depending on your use case, some metrics emitted by Dapr might contain values that have a high cardinality. This might cause increased memory usage for the Dapr process/container and incur expensive egress costs in certain cloud environments. To mitigate this issue, you can set regular expressions for every metric exposed by the Dapr sidecar. [See a list of all Dapr metrics](https://github.com/dapr/dapr/blob/master/docs/development/dapr-metrics.md).
|
||||
When invoking Dapr using HTTP, the legacy behavior (and current default as of Dapr 1.13) is to create a separate "bucket" for each requested method. When working with RESTful APIs, this can cause very high cardinality, with potential negative impact on memory usage and CPU.
|
||||
|
||||
Dapr 1.13 introduces a new option for the Dapr Configuration resource `spec.metrics.http.increasedCardinality`: when set to `false`, it reports metrics for the HTTP server for each "abstract" method (for example, requesting from a state store) instead of creating a "bucket" for each concrete request path.
|
||||
|
||||
The default value of `spec.metrics.http.increasedCardinality` is `true` in Dapr 1.13, to maintain the same behavior as Dapr 1.12 and older. However, the value will change to `false` (low-cardinality metrics by default) in Dapr 1.14.
|
||||
|
||||
Setting `spec.metrics.http.increasedCardinality` to `false` is **recommended** to all Dapr users, to reduce resource consumption. The pre-1.13 behavior, which is used when the option is `true`, is considered legacy and is only maintained for users who have special requirements around backwards-compatibility.
|
||||
|
||||
## Transform metrics with regular expressions
|
||||
|
||||
You can set regular expressions for every metric exposed by the Dapr sidecar to "transform" their values. [See a list of all Dapr metrics](https://github.com/dapr/dapr/blob/master/docs/development/dapr-metrics.md).
|
||||
|
||||
The following example shows how to apply a regular expression for the label `method` in the metric `dapr_runtime_service_invocation_req_sent_total`:
|
||||
|
||||
|
|
@ -84,9 +92,10 @@ kind: Configuration
|
|||
metadata:
|
||||
name: daprConfig
|
||||
spec:
|
||||
metric:
|
||||
enabled: true
|
||||
rules:
|
||||
metrics:
|
||||
enabled: true
|
||||
increasedCardinality: true
|
||||
rules:
|
||||
- name: dapr_runtime_service_invocation_req_sent_total
|
||||
labels:
|
||||
- name: method
|
||||
|
|
@ -96,12 +105,7 @@ spec:
|
|||
|
||||
When this configuration is applied, a recorded metric with the `method` label of `orders/a746dhsk293972nz` will be replaced with `orders/`.
|
||||
|
||||
### Watch the demo
|
||||
|
||||
Watch [this video to walk through handling high cardinality metrics](https://youtu.be/pOT8teL6j_k?t=1524):
|
||||
|
||||
<iframe width="560" height="315" src="https://www.youtube-nocookie.com/embed/pOT8teL6j_k?start=1524" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen></iframe>
|
||||
|
||||
Using regular expressions to reduce metrics cardinality is considered legacy. We encourage all users to set `spec.metrics.http.increasedCardinality` to `false` instead, which is simpler to configure and offers better performance.
|
||||
|
||||
## References
|
||||
|
||||
|
|
|
|||
Loading…
Reference in New Issue