mirror of https://github.com/dapr/docs.git
move observability
Signed-off-by: Hannah Hunter <hannahhunter@microsoft.com>
This commit is contained in:
Hannah Hunter
parent
cfaceb9207
commit
3448531c7f
|
|
@ -0,0 +1,14 @@
|
|||
---
|
||||
type: docs
|
||||
title: "Observability"
|
||||
linkTitle: "Observability"
|
||||
weight: 60
|
||||
description: See and measure the message calls to components and between networked services
|
||||
---
|
||||
|
||||
{{% alert title="More about Dapr Observability" color="primary" %}}
|
||||
Learn more about how to use Dapr Observability Lock:
|
||||
- Explore observability via any of the supporting [Dapr SDKs]({{< ref sdks >}}).
|
||||
- Review the [Observability API reference documentation]({{< ref health_api.md >}}).
|
||||
- Read the [general overview of the observability concept]({{< ref observability-concept >}}) in Dapr.
|
||||
{{% /alert %}}
|
||||
|
|
@ -0,0 +1,7 @@
|
|||
---
|
||||
type: docs
|
||||
title: "Health checks"
|
||||
linkTitle: "Health checks"
|
||||
weight: 100
|
||||
description: "How to setup health checks for the Dapr sidecar and your application"
|
||||
---
|
||||
|
|
@ -0,0 +1,169 @@
|
|||
---
|
||||
type: docs
|
||||
title: "App health checks"
|
||||
linkTitle: "App health checks"
|
||||
weight: 100
|
||||
description: Reacting to apps' health status changes
|
||||
---
|
||||
|
||||
The app health checks feature allows probing for the health of your application and reacting to status changes.
|
||||
|
||||
Applications can become unresponsive for a variety of reasons. For example, your application:
|
||||
- Could be too busy to accept new work;
|
||||
- Could have crashed; or
|
||||
- Could be in a deadlock state.
|
||||
|
||||
Sometimes the condition can be transitory, for example:
|
||||
- If the app is just busy and will resume accepting new work eventually
|
||||
- If the application is being restarted for whatever reason and is in its initialization phase
|
||||
|
||||
App health checks are disabled by default. Once you enable app health checks, the Dapr runtime (sidecar) periodically polls your application via HTTP or gRPC calls. When it detects a failure in the app's health, Dapr stops accepting new work on behalf of the application by:
|
||||
|
||||
- Unsubscribing from all pub/sub subscriptions
|
||||
- Stopping all input bindings
|
||||
- Short-circuiting all service-invocation requests, which terminate in the Dapr runtime and are not forwarded to the application
|
||||
|
||||
These changes are meant to be temporary, and Dapr resumes normal operations once it detects that the application is responsive again.
|
||||
|
||||
<img src="/images/observability-app-health.webp" width="800" alt="Diagram showing the app health feature. Running Dapr with app health enabled causes Dapr to periodically probe the app for its health.">
|
||||
|
||||
## App health checks vs platform-level health checks
|
||||
|
||||
App health checks in Dapr are meant to be complementary to, and not replace, any platform-level health checks, like [liveness probes](https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/) when running on Kubernetes.
|
||||
|
||||
Platform-level health checks (or liveness probes) generally ensure that the application is running, and cause the platform to restart the application in case of failures.
|
||||
|
||||
Unlike platform-level health checks, Dapr's app health checks focus on pausing work to an application that is currently unable to accept it, but is expected to be able to resume accepting work *eventually*. Goals include:
|
||||
|
||||
- Not bringing more load to an application that is already overloaded.
|
||||
- Do the "polite" thing by not taking messages from queues, bindings, or pub/sub brokers when Dapr knows the application won't be able to process them.
|
||||
|
||||
In this regard, Dapr's app health checks are "softer", waiting for an application to be able to process work, rather than terminating the running process in a "hard" way.
|
||||
|
||||
{{% alert title="Note" color="primary" %}}
|
||||
For Kubernetes, a failing app health check won't remove a pod from service discovery: this remains the responsibility of the Kubernetes liveness probe, _not_ Dapr.
|
||||
{{% /alert %}}
|
||||
|
||||
## Configuring app health checks
|
||||
|
||||
App health checks are disabled by default, but can be enabled with either:
|
||||
|
||||
- The `--enable-app-health-check` CLI flag; or
|
||||
- The `dapr.io/enable-app-health-check: true` annotation when running on Kubernetes.
|
||||
|
||||
Adding this flag is both necessary and sufficient to enable app health checks with the default options.
|
||||
|
||||
The full list of options are listed in this table:
|
||||
|
||||
| CLI flags | Kubernetes deployment annotation | Description | Default value |
|
||||
| ----------------------------- | ----------------------------------- | ----------- | ------------- |
|
||||
| `--enable-app-health-check` | `dapr.io/enable-app-health-check` | Boolean that enables the health checks | Disabled |
|
||||
| [`--app-health-check-path`]({{< ref "app-health.md#health-check-paths" >}}) | `dapr.io/app-health-check-path` | Path that Dapr invokes for health probes when the app channel is HTTP (this value is ignored if the app channel is using gRPC) | `/healthz` |
|
||||
| [`--app-health-probe-interval`]({{< ref "app-health.md#intervals-timeouts-and-thresholds" >}}) | `dapr.io/app-health-probe-interval` | Number of *seconds* between each health probe | `5` |
|
||||
| [`--app-health-probe-timeout`]({{< ref "app-health.md#intervals-timeouts-and-thresholds" >}}) | `dapr.io/app-health-probe-timeout` | Timeout in *milliseconds* for health probe requests | `500` |
|
||||
| [`--app-health-threshold`]({{< ref "app-health.md#intervals-timeouts-and-thresholds" >}}) | `dapr.io/app-health-threshold` | Max number of consecutive failures before the app is considered unhealthy | `3` |
|
||||
|
||||
> See the [full Dapr arguments and annotations reference]({{< ref arguments-annotations-overview >}}) for all options and how to enable them.
|
||||
|
||||
Additionally, app health checks are impacted by the protocol used for the app channel, which is configured with the following flag or annotation:
|
||||
|
||||
| CLI flag | Kubernetes deployment annotation | Description | Default value |
|
||||
| ----------------------------- | ----------------------------------- | ----------- | ------------- |
|
||||
| [`--app-protocol`]({{< ref "app-health.md#health-check-paths" >}}) | `dapr.io/app-protocol` | Protocol used for the app channel. supported values are `http`, `grpc`, `https`, `grpcs`, and `h2c` (HTTP/2 Cleartext). | `http` |
|
||||
|
||||
{{% alert title="Note" color="primary" %}}
|
||||
A low app health probe timeout value can classify an application as unhealthy if it experiences a sudden high load, causing the response time to degrade. If this happens, increase the `dapr.io/app-health-probe-timeout` value.
|
||||
{{% /alert %}}
|
||||
|
||||
### Health check paths
|
||||
|
||||
#### HTTP
|
||||
When using HTTP (including `http`, `https`, and `h2c`) for `app-protocol`, Dapr performs health probes by making an HTTP call to the path specified in `app-health-check-path`, which is `/health` by default.
|
||||
|
||||
For your app to be considered healthy, the response must have an HTTP status code in the 200-299 range. Any other status code is considered a failure. Dapr is only concerned with the status code of the response, and ignores any response header or body.
|
||||
|
||||
#### gRPC
|
||||
When using gRPC for the app channel (`app-protocol` set to `grpc` or `grpcs`), Dapr invokes the method `/dapr.proto.runtime.v1.AppCallbackHealthCheck/HealthCheck` in your application. Most likely, you will use a Dapr SDK to implement the handler for this method.
|
||||
|
||||
While responding to a health probe request, your app *may* decide to perform additional internal health checks to determine if it's ready to process work from the Dapr runtime. However, this is not required; it's a choice that depends on your application's needs.
|
||||
|
||||
### Intervals, timeouts, and thresholds
|
||||
|
||||
#### Intervals
|
||||
By default, when app health checks are enabled, Dapr probes your application every 5 seconds. You can configure the interval, in seconds, with `app-health-probe-interval`. These probes happen regularly, regardless of whether your application is healthy or not.
|
||||
|
||||
#### Timeouts
|
||||
When the Dapr runtime (sidecar) is initially started, Dapr waits for a successful health probe before considering the app healthy. This means that pub/sub subscriptions, input bindings, and service invocation requests won't be enabled for your application until this first health check is complete and successful.
|
||||
|
||||
Health probe requests are considered successful if the application sends a successful response (as explained above) within the timeout configured in `app-health-probe-timeout`. The default value is 500, corresponding to 500 milliseconds (half a second).
|
||||
|
||||
#### Thresholds
|
||||
Before Dapr considers an app to have entered an unhealthy state, it will wait for `app-health-threshold` consecutive failures, whose default value is 3. This default value means that your application must fail health probes 3 times *in a row* to be considered unhealthy.
|
||||
|
||||
If you set the threshold to 1, any failure causes Dapr to assume your app is unhealthy and will stop delivering work to it.
|
||||
|
||||
A threshold greater than 1 can help exclude transient failures due to external circumstances. The right value for your application depends on your requirements.
|
||||
|
||||
Thresholds only apply to failures. A single successful response is enough for Dapr to consider your app to be healthy and resume normal operations.
|
||||
|
||||
## Example
|
||||
|
||||
{{< tabs "Self-Hosted (CLI)" Kubernetes >}}
|
||||
|
||||
{{% codetab %}}
|
||||
|
||||
Use the CLI flags with the `dapr run` command to enable app health checks:
|
||||
|
||||
```sh
|
||||
dapr run \
|
||||
--app-id my-app \
|
||||
--app-port 7001 \
|
||||
--app-protocol http \
|
||||
--enable-app-health-check \
|
||||
--app-health-check-path=/healthz \
|
||||
--app-health-probe-interval 3 \
|
||||
--app-health-probe-timeout 200 \
|
||||
--app-health-threshold 2 \
|
||||
-- \
|
||||
<command to execute>
|
||||
```
|
||||
|
||||
{{% /codetab %}}
|
||||
|
||||
{{% codetab %}}
|
||||
|
||||
To enable app health checks in Kubernetes, add the relevant annotations to your Deployment:
|
||||
|
||||
```yaml
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: my-app
|
||||
labels:
|
||||
app: my-app
|
||||
spec:
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: my-app
|
||||
annotations:
|
||||
dapr.io/enabled: "true"
|
||||
dapr.io/app-id: "my-app"
|
||||
dapr.io/app-port: "7001"
|
||||
dapr.io/app-protocol: "http"
|
||||
dapr.io/enable-app-health-check: "true"
|
||||
dapr.io/app-health-check-path: "/healthz"
|
||||
dapr.io/app-health-probe-interval: "3"
|
||||
dapr.io/app-health-probe-timeout: "200"
|
||||
dapr.io/app-health-threshold: "2"
|
||||
```
|
||||
|
||||
{{% /codetab %}}
|
||||
|
||||
{{< /tabs >}}
|
||||
|
||||
## Demo
|
||||
|
||||
Watch this video for an [overview of using app health checks](https://youtu.be/srczBuOsAkI?t=533):
|
||||
|
||||
<iframe width="560" height="315" src="https://www.youtube-nocookie.com/embed/srczBuOsAkI?start=533" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
|
||||
|
|
@ -0,0 +1,101 @@
|
|||
---
|
||||
type: docs
|
||||
title: "Sidecar health"
|
||||
linkTitle: "Sidecar health"
|
||||
weight: 200
|
||||
description: Dapr sidecar health checks
|
||||
---
|
||||
|
||||
Dapr provides a way to determine its health using an [HTTP `/healthz` endpoint]({{< ref health_api.md >}}). With this endpoint, the *daprd* process, or sidecar, can be:
|
||||
|
||||
- Probed for its health
|
||||
- Determined for readiness and liveness
|
||||
|
||||
In this guide, you learn how the Dapr `/healthz` endpoint integrate with health probes from the application hosting platform (for example, Kubernetes).
|
||||
|
||||
When deploying Dapr to a hosting platform like Kubernetes, the Dapr health endpoint is automatically configured for you.
|
||||
|
||||
{{% alert title="Note" color="primary" %}}
|
||||
Dapr actors also have a health API endpoint where Dapr probes the application for a response to a signal from Dapr that the actor application is healthy and running. See [actor health API]({{< ref "actors_api.md#health-check" >}}).
|
||||
{{% /alert %}}
|
||||
|
||||
## Health endpoint: Integration with Kubernetes
|
||||
|
||||
Kubernetes uses *readiness* and *liveness* probes to determines the health of the container.
|
||||
|
||||
### Liveness
|
||||
The kubelet uses liveness probes to know when to restart a container. For example, liveness probes could catch a deadlock (a running application that is unable to make progress). Restarting a container in such a state can help to make the application more available despite having bugs.
|
||||
|
||||
#### How to configure a liveness probe in Kubernetes
|
||||
|
||||
In the pod configuration file, the liveness probe is added in the containers spec section as shown below:
|
||||
|
||||
```yaml
|
||||
livenessProbe:
|
||||
httpGet:
|
||||
path: /healthz
|
||||
port: 8080
|
||||
initialDelaySeconds: 3
|
||||
periodSeconds: 3
|
||||
```
|
||||
|
||||
In the above example, the `periodSeconds` field specifies that the kubelet should perform a liveness probe every 3 seconds. The `initialDelaySeconds` field tells the kubelet that it should wait 3 seconds before performing the first probe. To perform a probe, the kubelet sends an HTTP GET request to the server that is running in the container and listening on port 8080. If the handler for the server's `/healthz` path returns a success code, the kubelet considers the container to be alive and healthy. If the handler returns a failure code, the kubelet kills the container and restarts it.
|
||||
|
||||
Any HTTP status code between 200 and 399 indicates success; any other status code indicates failure.
|
||||
|
||||
### Readiness
|
||||
The kubelet uses readiness probes to know when a container is ready to start accepting traffic. A pod is considered ready when all of its containers are ready. One use of this readiness signal is to control which pods are used as backends for Kubernetes services. When a pod is not ready, it is removed from Kubernetes service load balancers.
|
||||
|
||||
{{% alert title="Note" color="primary" %}}
|
||||
The Dapr sidecar will be in ready state once the application is accessible on its configured port. The application cannot access the Dapr components during application start up/initialization.
|
||||
{{% /alert %}}
|
||||
|
||||
#### How to configure a readiness probe in Kubernetes
|
||||
|
||||
Readiness probes are configured similarly to liveness probes. The only difference is that you use the `readinessProbe` field instead of the `livenessProbe` field:
|
||||
|
||||
```yaml
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /healthz
|
||||
port: 8080
|
||||
initialDelaySeconds: 3
|
||||
periodSeconds: 3
|
||||
```
|
||||
|
||||
### Sidecar Injector
|
||||
|
||||
When integrating with Kubernetes, the Dapr sidecar is injected with a Kubernetes probe configuration telling it to use the Dapr `healthz` endpoint. This is done by the "Sidecar Injector" system service. The integration with the kubelet is shown in the diagram below.
|
||||
|
||||
<img src="/images/security-mTLS-dapr-system-services.png" width="800" alt="Diagram of Dapr services interacting" />
|
||||
|
||||
#### How the Dapr sidecar health endpoint is configured with Kubernetes
|
||||
|
||||
As mentioned above, this configuration is done automatically by the Sidecar Injector service. This section describes the specific values that are set on the liveness and readiness probes.
|
||||
|
||||
Dapr has its HTTP health endpoint `/v1.0/healthz` on port 3500. This can be used with Kubernetes for readiness and liveness probe. When the Dapr sidecar is injected, the readiness and liveness probes are configured in the pod configuration file with the following values:
|
||||
|
||||
```yaml
|
||||
livenessProbe:
|
||||
httpGet:
|
||||
path: v1.0/healthz
|
||||
port: 3500
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 10
|
||||
timeoutSeconds : 5
|
||||
failureThreshold : 3
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: v1.0/healthz
|
||||
port: 3500
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 10
|
||||
timeoutSeconds : 5
|
||||
failureThreshold: 3
|
||||
```
|
||||
|
||||
## Related links
|
||||
|
||||
- [Endpoint health API]({{< ref health_api.md >}})
|
||||
- [Actor health API]({{< ref "actors_api.md#health-check" >}})
|
||||
- [Kubernetes probe configuration parameters](https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/)
|
||||
|
|
@ -0,0 +1,8 @@
|
|||
---
|
||||
type: docs
|
||||
title: "Logging"
|
||||
linkTitle: "Logging"
|
||||
weight: 400
|
||||
description: "How to setup loggings for Dapr sidecar, and your application"
|
||||
---
|
||||
|
||||
|
|
@ -0,0 +1,189 @@
|
|||
---
|
||||
type: docs
|
||||
title: "How-To: Set up Fluentd, Elastic search and Kibana in Kubernetes"
|
||||
linkTitle: "FluentD"
|
||||
weight: 2000
|
||||
description: "How to install Fluentd, Elastic Search, and Kibana to search logs in Kubernetes"
|
||||
---
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Kubernetes (> 1.14)
|
||||
- [kubectl](https://kubernetes.io/docs/tasks/tools/)
|
||||
- [Helm 3](https://helm.sh/)
|
||||
|
||||
## Install Elastic search and Kibana
|
||||
|
||||
1. Create a Kubernetes namespace for monitoring tools
|
||||
|
||||
```bash
|
||||
kubectl create namespace dapr-monitoring
|
||||
```
|
||||
|
||||
2. Add the helm repo for Elastic Search
|
||||
|
||||
```bash
|
||||
helm repo add elastic https://helm.elastic.co
|
||||
helm repo update
|
||||
```
|
||||
|
||||
3. Install Elastic Search using Helm
|
||||
|
||||
By default, the chart creates 3 replicas which must be on different nodes. If your cluster has fewer than 3 nodes, specify a smaller number of replicas. For example, this sets the number of replicas to 1:
|
||||
|
||||
```bash
|
||||
helm install elasticsearch elastic/elasticsearch --version 7.17.3 -n dapr-monitoring --set replicas=1
|
||||
```
|
||||
|
||||
Otherwise:
|
||||
|
||||
```bash
|
||||
helm install elasticsearch elastic/elasticsearch --version 7.17.3 -n dapr-monitoring
|
||||
```
|
||||
|
||||
If you are using minikube or simply want to disable persistent volumes for development purposes, you can do so by using the following command:
|
||||
|
||||
```bash
|
||||
helm install elasticsearch elastic/elasticsearch --version 7.17.3 -n dapr-monitoring --set persistence.enabled=false,replicas=1
|
||||
```
|
||||
|
||||
4. Install Kibana
|
||||
|
||||
```bash
|
||||
helm install kibana elastic/kibana --version 7.17.3 -n dapr-monitoring
|
||||
```
|
||||
|
||||
5. Ensure that Elastic Search and Kibana are running in your Kubernetes cluster
|
||||
|
||||
```bash
|
||||
$ kubectl get pods -n dapr-monitoring
|
||||
NAME READY STATUS RESTARTS AGE
|
||||
elasticsearch-master-0 1/1 Running 0 6m58s
|
||||
kibana-kibana-95bc54b89-zqdrk 1/1 Running 0 4m21s
|
||||
```
|
||||
|
||||
## Install Fluentd
|
||||
|
||||
1. Install config map and Fluentd as a daemonset
|
||||
|
||||
Download these config files:
|
||||
- [fluentd-config-map.yaml](/docs/fluentd-config-map.yaml)
|
||||
- [fluentd-dapr-with-rbac.yaml](/docs/fluentd-dapr-with-rbac.yaml)
|
||||
|
||||
> Note: If you already have Fluentd running in your cluster, please enable the nested json parser so that it can parse JSON-formatted logs from Dapr.
|
||||
|
||||
Apply the configurations to your cluster:
|
||||
|
||||
```bash
|
||||
kubectl apply -f ./fluentd-config-map.yaml
|
||||
kubectl apply -f ./fluentd-dapr-with-rbac.yaml
|
||||
```
|
||||
|
||||
2. Ensure that Fluentd is running as a daemonset. The number of FluentD instances should be the same as the number of cluster nodes. In the example below, there is only one node in the cluster:
|
||||
|
||||
```bash
|
||||
$ kubectl get pods -n kube-system -w
|
||||
NAME READY STATUS RESTARTS AGE
|
||||
coredns-6955765f44-cxjxk 1/1 Running 0 4m41s
|
||||
coredns-6955765f44-jlskv 1/1 Running 0 4m41s
|
||||
etcd-m01 1/1 Running 0 4m48s
|
||||
fluentd-sdrld 1/1 Running 0 14s
|
||||
```
|
||||
|
||||
## Install Dapr with JSON formatted logs
|
||||
|
||||
1. Install Dapr with enabling JSON-formatted logs
|
||||
|
||||
```bash
|
||||
helm repo add dapr https://dapr.github.io/helm-charts/
|
||||
helm repo update
|
||||
helm install dapr dapr/dapr --namespace dapr-system --set global.logAsJson=true
|
||||
```
|
||||
|
||||
2. Enable JSON formatted log in Dapr sidecar
|
||||
|
||||
Add the `dapr.io/log-as-json: "true"` annotation to your deployment yaml. For example:
|
||||
|
||||
```yaml
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: pythonapp
|
||||
namespace: default
|
||||
labels:
|
||||
app: python
|
||||
spec:
|
||||
replicas: 1
|
||||
selector:
|
||||
matchLabels:
|
||||
app: python
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: python
|
||||
annotations:
|
||||
dapr.io/enabled: "true"
|
||||
dapr.io/app-id: "pythonapp"
|
||||
dapr.io/log-as-json: "true"
|
||||
...
|
||||
```
|
||||
|
||||
## Search logs
|
||||
|
||||
> Note: Elastic Search takes a time to index the logs that Fluentd sends.
|
||||
|
||||
1. Port-forward from localhost to `svc/kibana-kibana`
|
||||
|
||||
```bash
|
||||
$ kubectl port-forward svc/kibana-kibana 5601 -n dapr-monitoring
|
||||
Forwarding from 127.0.0.1:5601 -> 5601
|
||||
Forwarding from [::1]:5601 -> 5601
|
||||
Handling connection for 5601
|
||||
Handling connection for 5601
|
||||
```
|
||||
|
||||
2. Browse to `http://localhost:5601`
|
||||
|
||||
3. Expand the drop-down menu and click **Management → Stack Management**
|
||||
|
||||
|
||||
|
||||
4. On the Stack Management page, select **Data → Index Management** and wait until `dapr-*` is indexed.
|
||||
|
||||
|
||||
|
||||
5. Once `dapr-*` is indexed, click on **Kibana → Index Patterns** and then the **Create index pattern** button.
|
||||
|
||||
|
||||
|
||||
6. Define a new index pattern by typing `dapr*` into the **Index Pattern name** field, then click the **Next step** button to continue.
|
||||
|
||||
|
||||
|
||||
7. Configure the primary time field to use with the new index pattern by selecting the `@timestamp` option from the **Time field** drop-down. Click the **Create index pattern** button to complete creation of the index pattern.
|
||||
|
||||
|
||||
|
||||
8. The newly created index pattern should be shown. Confirm that the fields of interest such as `scope`, `type`, `app_id`, `level`, etc. are being indexed by using the search box in the **Fields** tab.
|
||||
|
||||
> Note: If you cannot find the indexed field, please wait. The time it takes to search across all indexed fields depends on the volume of data and size of the resource that the elastic search is running on.
|
||||
|
||||
|
||||
|
||||
9. To explore the indexed data, expand the drop-down menu and click **Analytics → Discover**.
|
||||
|
||||
|
||||
|
||||
10. In the search box, type in a query string such as `scope:*` and click the **Refresh** button to view the results.
|
||||
|
||||
> Note: This can take a long time. The time it takes to return all results depends on the volume of data and size of the resource that the elastic search is running on.
|
||||
|
||||
|
||||
|
||||
## References
|
||||
|
||||
* [Fluentd for Kubernetes](https://docs.fluentd.org/v/0.12/articles/kubernetes-fluentd)
|
||||
* [Elastic search helm chart](https://github.com/elastic/helm-charts/tree/master/elasticsearch)
|
||||
* [Kibana helm chart](https://github.com/elastic/helm-charts/tree/master/kibana)
|
||||
* [Kibana Query Language](https://www.elastic.co/guide/en/kibana/current/kuery-query.html)
|
||||
* [Troubleshooting using Logs]({{< ref "logs-troubleshooting.md" >}})
|
||||
|
|
@ -0,0 +1,133 @@
|
|||
---
|
||||
type: docs
|
||||
title: "Logs"
|
||||
linkTitle: "Logs"
|
||||
weight: 1000
|
||||
description: "Understand Dapr logging"
|
||||
---
|
||||
|
||||
Dapr produces structured logs to stdout, either in plain-text or JSON-formatted. By default, all Dapr processes (runtime, or sidecar, and all control plane services) write logs to the console (stdout) in plain-text. To enable JSON-formatted logging, you need to add the `--log-as-json` command flag when running Dapr processes.
|
||||
|
||||
{{% alert title="Note" color="primary" %}}
|
||||
If you want to use a search engine such as Elastic Search or Azure Monitor to search the logs, it is strongly recommended to use JSON-formatted logs which the log collector and search engine can parse using the built-in JSON parser.
|
||||
{{% /alert %}}
|
||||
|
||||
## Log schema
|
||||
|
||||
Dapr produces logs based on the following schema:
|
||||
|
||||
| Field | Description | Example |
|
||||
|-------|-------------------|---------|
|
||||
| time | ISO8601 Timestamp | `2011-10-05T14:48:00.000Z` |
|
||||
| level | Log Level (info/warn/debug/error) | `info` |
|
||||
| type | Log Type | `log` |
|
||||
| msg | Log Message | `hello dapr!` |
|
||||
| scope | Logging Scope | `dapr.runtime` |
|
||||
| instance | Container Name | `dapr-pod-xxxxx` |
|
||||
| app_id | Dapr App ID | `dapr-app` |
|
||||
| ver | Dapr Runtime Version | `1.9.0` |
|
||||
|
||||
API logging may add other structured fields, as described in the [documentation for API logging]({{< ref "api-logs-troubleshooting.md" >}}).
|
||||
|
||||
## Plain text and JSON formatted logs
|
||||
|
||||
* Plain-text log examples
|
||||
|
||||
```bash
|
||||
time="2022-11-01T17:08:48.303776-07:00" level=info msg="starting Dapr Runtime -- version 1.9.0 -- commit v1.9.0-g5dfcf2e" instance=dapr-pod-xxxx scope=dapr.runtime type=log ver=1.9.0
|
||||
time="2022-11-01T17:08:48.303913-07:00" level=info msg="log level set to: info" instance=dapr-pod-xxxx scope=dapr.runtime type=log ver=1.9.0
|
||||
```
|
||||
|
||||
* JSON-formatted log examples
|
||||
|
||||
```json
|
||||
{"instance":"dapr-pod-xxxx","level":"info","msg":"starting Dapr Runtime -- version 1.9.0 -- commit v1.9.0-g5dfcf2e","scope":"dapr.runtime","time":"2022-11-01T17:09:45.788005Z","type":"log","ver":"1.9.0"}
|
||||
{"instance":"dapr-pod-xxxx","level":"info","msg":"log level set to: info","scope":"dapr.runtime","time":"2022-11-01T17:09:45.788075Z","type":"log","ver":"1.9.0"}
|
||||
```
|
||||
|
||||
## Log formats
|
||||
|
||||
Dapr supports printing either plain-text, the default, or JSON-formatted logs.
|
||||
|
||||
To use JSON-formatted logs, you need to add additional configuration options when you install Dapr and when deploy your apps. The recommendation is to use JSON-formatted logs because most log collectors and search engines can parse JSON more easily with built-in parsers.
|
||||
|
||||
## Enabling JSON logging with the Dapr CLI
|
||||
|
||||
When using the Dapr CLI to run an application, pass the `--log-as-json` option to enable JSON-formatted logs, for example:
|
||||
|
||||
```sh
|
||||
dapr run \
|
||||
--app-id orderprocessing \
|
||||
--resources-path ./components/ \
|
||||
--log-as-json \
|
||||
-- python3 OrderProcessingService.py
|
||||
```
|
||||
|
||||
## Enabling JSON logging in Kubernetes
|
||||
|
||||
The following steps describe how to configure JSON-formatted logs for Kubernetes
|
||||
|
||||
### Dapr control plane
|
||||
|
||||
All services in the Dapr control plane (such as `operator`, `sentry`, etc) support a `--log-as-json` option to enable JSON-formatted logging.
|
||||
|
||||
If you're deploying Dapr to Kubernetes using a Helm chart, you can enable JSON-formatted logs for Dapr system services by passing the `--set global.logAsJson=true` option; for example:
|
||||
|
||||
```bash
|
||||
helm upgrade --install dapr \
|
||||
dapr/dapr \
|
||||
--namespace dapr-system \
|
||||
--set global.logAsJson=true
|
||||
```
|
||||
|
||||
### Enable JSON-formatted log for Dapr sidecars
|
||||
|
||||
You can enable JSON-formatted logs in Dapr sidecars by adding the `dapr.io/log-as-json: "true"` annotation to the deployment, for example:
|
||||
|
||||
```yaml
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: pythonapp
|
||||
labels:
|
||||
app: python
|
||||
spec:
|
||||
selector:
|
||||
matchLabels:
|
||||
app: python
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: python
|
||||
annotations:
|
||||
dapr.io/enabled: "true"
|
||||
dapr.io/app-id: "pythonapp"
|
||||
# This enables JSON-formatted logging
|
||||
dapr.io/log-as-json: "true"
|
||||
...
|
||||
```
|
||||
|
||||
## API Logging
|
||||
|
||||
API logging enables you to see the API calls your application makes to the Dapr sidecar, to debug issues or monitor the behavior of your application. You can combine both Dapr API logging with Dapr log events.
|
||||
|
||||
See [configure and view Dapr Logs]({{< ref "logs-troubleshooting.md" >}}) and [configure and view Dapr API Logs]({{< ref "api-logs-troubleshooting.md" >}}) for more information.
|
||||
|
||||
## Log collectors
|
||||
|
||||
If you run Dapr in a Kubernetes cluster, [Fluentd](https://www.fluentd.org/) is a popular container log collector. You can use Fluentd with a [JSON parser plugin](https://docs.fluentd.org/parser/json) to parse Dapr JSON-formatted logs. This [how-to]({{< ref fluentd.md >}}) shows how to configure Fluentd in your cluster.
|
||||
|
||||
If you are using Azure Kubernetes Service, you can use the built-in agent to collect logs with Azure Monitor without needing to install Fluentd.
|
||||
|
||||
## Search engines
|
||||
|
||||
If you use [Fluentd](https://www.fluentd.org/), we recommend using Elastic Search and Kibana. This [how-to]({{< ref fluentd.md >}}) shows how to set up Elastic Search and Kibana in your Kubernetes cluster.
|
||||
|
||||
If you are using the Azure Kubernetes Service, you can use [Azure Monitor for containers](https://docs.microsoft.com/azure/azure-monitor/insights/container-insights-overview) without installing any additional monitoring tools. Also read [How to enable Azure Monitor for containers](https://docs.microsoft.com/azure/azure-monitor/insights/container-insights-onboard)
|
||||
|
||||
## References
|
||||
|
||||
- [How-to: Set up Fleuntd, Elastic search, and Kibana]({{< ref fluentd.md >}})
|
||||
- [How-to: Set up Azure Monitor in Azure Kubernetes Service]({{< ref azure-monitor.md >}})
|
||||
- [Configure and view Dapr Logs]({{< ref "logs-troubleshooting.md" >}})
|
||||
- [Configure and view Dapr API Logs]({{< ref "api-logs-troubleshooting.md" >}})
|
||||
|
|
@ -0,0 +1,78 @@
|
|||
---
|
||||
type: docs
|
||||
title: "How-To: Set-up New Relic for Dapr logging"
|
||||
linkTitle: "New Relic"
|
||||
weight: 3000
|
||||
description: "Set-up New Relic for Dapr logging"
|
||||
---
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Perpetually [free New Relic account](https://newrelic.com/signup?ref=dapr), 100 GB/month of free data ingest, 1 free full access user, unlimited free basic users
|
||||
|
||||
## Background
|
||||
|
||||
New Relic offers a [Fluent Bit](https://fluentbit.io/) output [plugin](https://github.com/newrelic/newrelic-fluent-bit-output) to easily forward your logs to [New Relic Logs](https://github.com/newrelic/newrelic-fluent-bit-output). This plugin is also provided in a standalone Docker image that can be installed in a Kubernetes cluster in the form of a DaemonSet, which we refer as the Kubernetes plugin.
|
||||
|
||||
This document explains how to install it in your cluster, either using a Helm chart (recommended), or manually by applying Kubernetes manifests.
|
||||
|
||||
## Installation
|
||||
|
||||
### Install using the Helm chart (recommended)
|
||||
|
||||
1. Install Helm following the official instructions.
|
||||
|
||||
2. Add the New Relic official Helm chart repository following these instructions
|
||||
|
||||
3. Run the following command to install the New Relic Logging Kubernetes plugin via Helm, replacing the placeholder value YOUR_LICENSE_KEY with your [New Relic license key](https://docs.newrelic.com/docs/accounts/accounts-billing/account-setup/new-relic-license-key/):
|
||||
|
||||
- Helm 3
|
||||
```bash
|
||||
helm install newrelic-logging newrelic/newrelic-logging --set licenseKey=YOUR_LICENSE_KEY
|
||||
```
|
||||
|
||||
- Helm 2
|
||||
```bash
|
||||
helm install newrelic/newrelic-logging --name newrelic-logging --set licenseKey=YOUR_LICENSE_KEY
|
||||
```
|
||||
|
||||
For EU users, add `--set endpoint=https://log-api.eu.newrelic.com/log/v1 to any of the helm install commands above.
|
||||
|
||||
By default, tailing is set to /var/log/containers/*.log. To change this setting, provide your preferred path by adding --set fluentBit.path=DESIRED_PATH to any of the helm install commands above.
|
||||
|
||||
### Install the Kubernetes manifest
|
||||
|
||||
1. Download the following 3 manifest files into your current working directory:
|
||||
|
||||
```bash
|
||||
curl https://raw.githubusercontent.com/newrelic/helm-charts/master/charts/newrelic-logging/k8s/fluent-conf.yml > fluent-conf.yml
|
||||
curl https://raw.githubusercontent.com/newrelic/helm-charts/master/charts/newrelic-logging/k8s/new-relic-fluent-plugin.yml > new-relic-fluent-plugin.yml
|
||||
curl https://raw.githubusercontent.com/newrelic/helm-charts/master/charts/newrelic-logging/k8s/rbac.yml > rbac.yml
|
||||
```
|
||||
|
||||
2. In the downloaded new-relic-fluent-plugin.yml file, replace the placeholder value LICENSE_KEY with your New Relic license key.
|
||||
|
||||
For EU users, replace the ENDPOINT environment variable to https://log-api.eu.newrelic.com/log/v1.
|
||||
|
||||
3. Once the License key has been added, run the following command in your terminal or command-line interface:
|
||||
```bash
|
||||
kubectl apply -f .
|
||||
```
|
||||
|
||||
4. [OPTIONAL] You can configure how the plugin parses the data by editing the parsers.conf section in the fluent-conf.yml file. For more information, see Fluent Bit's documentation on Parsers configuration.
|
||||
|
||||
By default, tailing is set to /var/log/containers/*.log. To change this setting, replace the default path with your preferred path in the new-relic-fluent-plugin.yml file.
|
||||
|
||||
## View Logs
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
## Related Links/References
|
||||
|
||||
* [New Relic Account Signup](https://newrelic.com/signup)
|
||||
* [Telemetry Data Platform](https://newrelic.com/platform/telemetry-data-platform)
|
||||
* [New Relic Logging](https://github.com/newrelic/helm-charts/tree/master/charts/newrelic-logging)
|
||||
* [Types of New Relic API keys](https://docs.newrelic.com/docs/apis/intro-apis/new-relic-api-keys/)
|
||||
* [Alerts and Applied Intelligence](https://docs.newrelic.com/docs/alerts-applied-intelligence/overview/)
|
||||
|
|
@ -0,0 +1,7 @@
|
|||
---
|
||||
type: docs
|
||||
title: "Metrics"
|
||||
linkTitle: "Metrics"
|
||||
weight: 300
|
||||
description: "How to view Dapr metrics"
|
||||
---
|
||||
|
|
@ -0,0 +1,134 @@
|
|||
---
|
||||
type: docs
|
||||
title: "How-To: Set up Azure Monitor to search logs and collect metrics"
|
||||
linkTitle: "Azure Monitor"
|
||||
weight: 7000
|
||||
description: "Enable Dapr metrics and logs with Azure Monitor for Azure Kubernetes Service (AKS)"
|
||||
---
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- [Azure Kubernetes Service](https://docs.microsoft.com/azure/aks/)
|
||||
- [Enable Azure Monitor For containers in AKS](https://docs.microsoft.com/azure/azure-monitor/insights/container-insights-overview)
|
||||
- [kubectl](https://kubernetes.io/docs/tasks/tools/)
|
||||
- [Helm 3](https://helm.sh/)
|
||||
|
||||
## Enable Prometheus metric scrape using config map
|
||||
|
||||
1. Make sure that Azure Monitor Agents (AMA) are running.
|
||||
|
||||
```bash
|
||||
$ kubectl get pods -n kube-system
|
||||
NAME READY STATUS RESTARTS AGE
|
||||
...
|
||||
ama-logs-48kpv 2/2 Running 0 2d13h
|
||||
ama-logs-mx24c 2/2 Running 0 2d13h
|
||||
ama-logs-rs-f9bbb9898-vbt6k 1/1 Running 0 30h
|
||||
ama-logs-sm2mz 2/2 Running 0 2d13h
|
||||
ama-logs-z7p4c 2/2 Running 0 2d13h
|
||||
...
|
||||
```
|
||||
|
||||
1. Apply config map to enable Prometheus metrics endpoint scrape.
|
||||
|
||||
You can use [azm-config-map.yaml](/docs/azm-config-map.yaml) to enable Prometheus metrics endpoint scrape.
|
||||
|
||||
If you installed Dapr to a different namespace, you need to change the `monitor_kubernetes_pod_namespaces` array values. For example:
|
||||
|
||||
```yaml
|
||||
...
|
||||
prometheus-data-collection-settings: |-
|
||||
[prometheus_data_collection_settings.cluster]
|
||||
interval = "1m"
|
||||
monitor_kubernetes_pods = true
|
||||
monitor_kubernetes_pods_namespaces = ["dapr-system", "default"]
|
||||
[prometheus_data_collection_settings.node]
|
||||
interval = "1m"
|
||||
...
|
||||
```
|
||||
|
||||
Apply config map:
|
||||
|
||||
```bash
|
||||
kubectl apply -f ./azm-config.map.yaml
|
||||
```
|
||||
|
||||
## Install Dapr with JSON formatted logs
|
||||
|
||||
1. Install Dapr with enabling JSON-formatted logs.
|
||||
|
||||
```bash
|
||||
helm install dapr dapr/dapr --namespace dapr-system --set global.logAsJson=true
|
||||
```
|
||||
|
||||
1. Enable JSON formatted log in Dapr sidecar and add Prometheus annotations.
|
||||
|
||||
> Note: The Azure Monitor Agents (AMA) only sends the metrics if the Prometheus annotations are set.
|
||||
|
||||
Add `dapr.io/log-as-json: "true"` annotation to your deployment yaml.
|
||||
|
||||
Example:
|
||||
|
||||
```yaml
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: pythonapp
|
||||
namespace: default
|
||||
labels:
|
||||
app: python
|
||||
spec:
|
||||
replicas: 1
|
||||
selector:
|
||||
matchLabels:
|
||||
app: python
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: python
|
||||
annotations:
|
||||
dapr.io/enabled: "true"
|
||||
dapr.io/app-id: "pythonapp"
|
||||
dapr.io/log-as-json: "true"
|
||||
prometheus.io/scrape: "true"
|
||||
prometheus.io/port: "9090"
|
||||
prometheus.io/path: "/"
|
||||
|
||||
...
|
||||
```
|
||||
|
||||
## Search metrics and logs with Azure Monitor
|
||||
|
||||
1. Go to Azure Monitor in the Azure portal.
|
||||
|
||||
1. Search Dapr **Logs**.
|
||||
|
||||
Here is an example query, to parse JSON formatted logs and query logs from Dapr system processes.
|
||||
|
||||
```
|
||||
ContainerLog
|
||||
| extend parsed=parse_json(LogEntry)
|
||||
| project Time=todatetime(parsed['time']), app_id=parsed['app_id'], scope=parsed['scope'],level=parsed['level'], msg=parsed['msg'], type=parsed['type'], ver=parsed['ver'], instance=parsed['instance']
|
||||
| where level != ""
|
||||
| sort by Time
|
||||
```
|
||||
|
||||
1. Search **Metrics**.
|
||||
|
||||
This query, queries `process_resident_memory_bytes` Prometheus metrics for Dapr system processes and renders timecharts.
|
||||
|
||||
```
|
||||
InsightsMetrics
|
||||
| where Namespace == "prometheus" and Name == "process_resident_memory_bytes"
|
||||
| extend tags=parse_json(Tags)
|
||||
| project TimeGenerated, Name, Val, app=tostring(tags['app'])
|
||||
| summarize memInBytes=percentile(Val, 99) by bin(TimeGenerated, 1m), app
|
||||
| where app startswith "dapr-"
|
||||
| render timechart
|
||||
```
|
||||
|
||||
## References
|
||||
|
||||
- [Configure scraping of Prometheus metrics with Azure Monitor for containers](https://docs.microsoft.com/azure/azure-monitor/insights/container-insights-prometheus-integration)
|
||||
- [Configure agent data collection for Azure Monitor for containers](https://docs.microsoft.com/azure/azure-monitor/insights/container-insights-agent-config)
|
||||
- [Azure Monitor Query](https://docs.microsoft.com/azure/azure-monitor/log-query/query-language)
|
||||
|
|
@ -0,0 +1,182 @@
|
|||
---
|
||||
type: docs
|
||||
title: "How-To: Observe metrics with Grafana"
|
||||
linkTitle: "Grafana dashboards"
|
||||
weight: 5000
|
||||
description: "How to view Dapr metrics in a Grafana dashboard."
|
||||
---
|
||||
|
||||
## Available dashboards
|
||||
|
||||
{{< tabs "System Service" "Sidecars" "Actors" >}}
|
||||
|
||||
{{% codetab %}}
|
||||
The `grafana-system-services-dashboard.json` template shows Dapr system component status, dapr-operator, dapr-sidecar-injector, dapr-sentry, and dapr-placement:
|
||||
|
||||
<img src="/images/grafana-system-service-dashboard.png" alt="Screenshot of the system service dashboard" width=1200>
|
||||
{{% /codetab %}}
|
||||
|
||||
{{% codetab %}}
|
||||
The `grafana-sidecar-dashboard.json` template shows Dapr sidecar status, including sidecar health/resources, throughput/latency of HTTP and gRPC, Actor, mTLS, etc.:
|
||||
|
||||
<img src="/images/grafana-sidecar-dashboard.png" alt="Screenshot of the sidecar dashboard" width=1200>
|
||||
{{% /codetab %}}
|
||||
|
||||
{{% codetab %}}
|
||||
The `grafana-actor-dashboard.json` template shows Dapr Sidecar status, actor invocation throughput/latency, timer/reminder triggers, and turn-based concurrnecy:
|
||||
|
||||
<img src="/images/grafana-actor-dashboard.png" alt="Screenshot of the actor dashboard" width=1200>
|
||||
{{% /codetab %}}
|
||||
|
||||
{{< /tabs >}}
|
||||
|
||||
## Pre-requisites
|
||||
|
||||
- [Setup Prometheus]({{<ref prometheus.md>}})
|
||||
|
||||
## Setup on Kubernetes
|
||||
|
||||
### Install Grafana
|
||||
|
||||
1. Add the Grafana Helm repo:
|
||||
|
||||
```bash
|
||||
helm repo add grafana https://grafana.github.io/helm-charts
|
||||
helm repo update
|
||||
```
|
||||
|
||||
1. Install the chart:
|
||||
|
||||
```bash
|
||||
helm install grafana grafana/grafana -n dapr-monitoring
|
||||
```
|
||||
|
||||
{{% alert title="Note" color="primary" %}}
|
||||
If you are Minikube user or want to disable persistent volume for development purpose, you can disable it by using the following command instead:
|
||||
|
||||
```bash
|
||||
helm install grafana grafana/grafana -n dapr-monitoring --set persistence.enabled=false
|
||||
```
|
||||
{{% /alert %}}
|
||||
|
||||
|
||||
1. Retrieve the admin password for Grafana login:
|
||||
|
||||
```bash
|
||||
kubectl get secret --namespace dapr-monitoring grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo
|
||||
```
|
||||
|
||||
You will get a password similar to `cj3m0OfBNx8SLzUlTx91dEECgzRlYJb60D2evof1%`. Remove the `%` character from the password to get `cj3m0OfBNx8SLzUlTx91dEECgzRlYJb60D2evof1` as the admin password.
|
||||
|
||||
1. Validation Grafana is running in your cluster:
|
||||
|
||||
```bash
|
||||
kubectl get pods -n dapr-monitoring
|
||||
|
||||
NAME READY STATUS RESTARTS AGE
|
||||
dapr-prom-kube-state-metrics-9849d6cc6-t94p8 1/1 Running 0 4m58s
|
||||
dapr-prom-prometheus-alertmanager-749cc46f6-9b5t8 2/2 Running 0 4m58s
|
||||
dapr-prom-prometheus-node-exporter-5jh8p 1/1 Running 0 4m58s
|
||||
dapr-prom-prometheus-node-exporter-88gbg 1/1 Running 0 4m58s
|
||||
dapr-prom-prometheus-node-exporter-bjp9f 1/1 Running 0 4m58s
|
||||
dapr-prom-prometheus-pushgateway-688665d597-h4xx2 1/1 Running 0 4m58s
|
||||
dapr-prom-prometheus-server-694fd8d7c-q5d59 2/2 Running 0 4m58s
|
||||
grafana-c49889cff-x56vj 1/1 Running 0 5m10s
|
||||
```
|
||||
|
||||
### Configure Prometheus as data source
|
||||
First you need to connect Prometheus as a data source to Grafana.
|
||||
|
||||
1. Port-forward to svc/grafana:
|
||||
|
||||
```bash
|
||||
kubectl port-forward svc/grafana 8080:80 -n dapr-monitoring
|
||||
|
||||
Forwarding from 127.0.0.1:8080 -> 3000
|
||||
Forwarding from [::1]:8080 -> 3000
|
||||
Handling connection for 8080
|
||||
Handling connection for 8080
|
||||
```
|
||||
|
||||
1. Open a browser to `http://localhost:8080`
|
||||
|
||||
1. Login to Grafana
|
||||
- Username = `admin`
|
||||
- Password = Password from above
|
||||
|
||||
1. Select `Configuration` and `Data Sources`
|
||||
|
||||
<img src="/images/grafana-datasources.png" alt="Screenshot of the Grafana add Data Source menu" width=200>
|
||||
|
||||
|
||||
1. Add Prometheus as a data source.
|
||||
|
||||
<img src="/images/grafana-add-datasources.png" alt="Screenshot of the Prometheus add Data Source" width=600>
|
||||
|
||||
1. Get your Prometheus HTTP URL
|
||||
|
||||
The Prometheus HTTP URL follows the format `http://<prometheus service endpoint>.<namespace>`
|
||||
|
||||
Start by getting the Prometheus server endpoint by running the following command:
|
||||
|
||||
```bash
|
||||
kubectl get svc -n dapr-monitoring
|
||||
|
||||
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
|
||||
dapr-prom-kube-state-metrics ClusterIP 10.0.174.177 <none> 8080/TCP 7d9h
|
||||
dapr-prom-prometheus-alertmanager ClusterIP 10.0.255.199 <none> 80/TCP 7d9h
|
||||
dapr-prom-prometheus-node-exporter ClusterIP None <none> 9100/TCP 7d9h
|
||||
dapr-prom-prometheus-pushgateway ClusterIP 10.0.190.59 <none> 9091/TCP 7d9h
|
||||
dapr-prom-prometheus-server ClusterIP 10.0.172.191 <none> 80/TCP 7d9h
|
||||
elasticsearch-master ClusterIP 10.0.36.146 <none> 9200/TCP,9300/TCP 7d10h
|
||||
elasticsearch-master-headless ClusterIP None <none> 9200/TCP,9300/TCP 7d10h
|
||||
grafana ClusterIP 10.0.15.229 <none> 80/TCP 5d5h
|
||||
kibana-kibana ClusterIP 10.0.188.224 <none> 5601/TCP 7d10h
|
||||
|
||||
```
|
||||
|
||||
In this guide the server name is `dapr-prom-prometheus-server` and the namespace is `dapr-monitoring`, so the HTTP URL will be `http://dapr-prom-prometheus-server.dapr-monitoring`.
|
||||
|
||||
1. Fill in the following settings:
|
||||
|
||||
- Name: `Dapr`
|
||||
- HTTP URL: `http://dapr-prom-prometheus-server.dapr-monitoring`
|
||||
- Default: On
|
||||
- Skip TLS Verify: On
|
||||
- Necessary in order to save and test the configuration
|
||||
|
||||
<img src="/images/grafana-prometheus-dapr-server-url.png" alt="Screenshot of the Prometheus Data Source configuration" width=600>
|
||||
|
||||
1. Click `Save & Test` button to verify that the connection succeeded.
|
||||
|
||||
## Import dashboards in Grafana
|
||||
|
||||
1. In the upper left corner of the Grafana home screen, click the "+" option, then "Import".
|
||||
|
||||
You can now import [Grafana dashboard templates](https://github.com/dapr/dapr/tree/master/grafana) from [release assets](https://github.com/dapr/dapr/releases) for your Dapr version:
|
||||
|
||||
<img src="/images/grafana-uploadjson.png" alt="Screenshot of the Grafana dashboard upload option" width=700>
|
||||
|
||||
1. Find the dashboard that you imported and enjoy
|
||||
|
||||
<img src="/images/system-service-dashboard.png" alt="Screenshot of Dapr service dashboard" width=900>
|
||||
|
||||
{{% alert title="Tip" color="primary" %}}
|
||||
Hover your mouse over the `i` in the corner to the description of each chart:
|
||||
|
||||
<img src="/images/grafana-tooltip.png" alt="Screenshot of the tooltip for graphs" width=700>
|
||||
{{% /alert %}}
|
||||
|
||||
## References
|
||||
|
||||
* [Dapr Observability]({{<ref observability-concept.md >}})
|
||||
* [Prometheus Installation](https://github.com/prometheus-community/helm-charts)
|
||||
* [Prometheus on Kubernetes](https://github.com/coreos/kube-prometheus)
|
||||
* [Prometheus Query Language](https://prometheus.io/docs/prometheus/latest/querying/basics/)
|
||||
* [Supported Dapr metrics](https://github.com/dapr/dapr/blob/master/docs/development/dapr-metrics.md)
|
||||
|
||||
## Example
|
||||
|
||||
<div class="embed-responsive embed-responsive-16by9">
|
||||
<iframe width="560" height="315" src="https://www.youtube-nocookie.com/embed/8W-iBDNvCUM?start=2577" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
|
||||
</div>
|
||||
|
|
@ -0,0 +1,109 @@
|
|||
---
|
||||
type: docs
|
||||
title: "Configure metrics"
|
||||
linkTitle: "Configure metrics"
|
||||
weight: 4000
|
||||
description: "Enable or disable Dapr metrics "
|
||||
---
|
||||
|
||||
By default, each Dapr system process emits Go runtime/process metrics and has their own [Dapr metrics](https://github.com/dapr/dapr/blob/master/docs/development/dapr-metrics.md).
|
||||
|
||||
## Prometheus endpoint
|
||||
The Dapr sidecars exposes a [Prometheus](https://prometheus.io/) metrics endpoint that you can scrape to gain a greater understanding of how Dapr is behaving.
|
||||
|
||||
## Configuring metrics using the CLI
|
||||
|
||||
The metrics application endpoint is enabled by default. You can disable it by passing the command line argument `--enable-metrics=false`.
|
||||
|
||||
The default metrics port is `9090`. You can override this by passing the command line argument `--metrics-port` to Daprd.
|
||||
|
||||
## Configuring metrics in Kubernetes
|
||||
You can also enable/disable the metrics for a specific application by setting the `dapr.io/enable-metrics: "false"` annotation on your application deployment. With the metrics exporter disabled, `daprd` does not open the metrics listening port.
|
||||
|
||||
The following Kubernetes deployment example shows how metrics are explicitly enabled with the port specified as "9090".
|
||||
|
||||
```yaml
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: nodeapp
|
||||
labels:
|
||||
app: node
|
||||
spec:
|
||||
replicas: 1
|
||||
selector:
|
||||
matchLabels:
|
||||
app: node
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: node
|
||||
annotations:
|
||||
dapr.io/enabled: "true"
|
||||
dapr.io/app-id: "nodeapp"
|
||||
dapr.io/app-port: "3000"
|
||||
dapr.io/enable-metrics: "true"
|
||||
dapr.io/metrics-port: "9090"
|
||||
spec:
|
||||
containers:
|
||||
- name: node
|
||||
image: dapriosamples/hello-k8s-node:latest
|
||||
ports:
|
||||
- containerPort: 3000
|
||||
imagePullPolicy: Always
|
||||
```
|
||||
|
||||
## Configuring metrics using application configuration
|
||||
You can also enable metrics via application configuration. To disable the metrics collection in the Dapr sidecars running in a specific namespace:
|
||||
|
||||
- Use the `metrics` spec configuration.
|
||||
- Set `enabled: false` to disable the metrics in the Dapr runtime.
|
||||
|
||||
```yaml
|
||||
apiVersion: dapr.io/v1alpha1
|
||||
kind: Configuration
|
||||
metadata:
|
||||
name: tracing
|
||||
namespace: default
|
||||
spec:
|
||||
tracing:
|
||||
samplingRate: "1"
|
||||
metrics:
|
||||
enabled: false
|
||||
```
|
||||
|
||||
## High cardinality metrics
|
||||
|
||||
Depending on your use case, some metrics emitted by Dapr might contain values that have a high cardinality. This might cause increased memory usage for the Dapr process/container and incur expensive egress costs in certain cloud environments. To mitigate this issue, you can set regular expressions for every metric exposed by the Dapr sidecar. [See a list of all Dapr metrics](https://github.com/dapr/dapr/blob/master/docs/development/dapr-metrics.md).
|
||||
|
||||
The following example shows how to apply a regular expression for the label `method` in the metric `dapr_runtime_service_invocation_req_sent_total`:
|
||||
|
||||
```yaml
|
||||
apiVersion: dapr.io/v1alpha1
|
||||
kind: Configuration
|
||||
metadata:
|
||||
name: daprConfig
|
||||
spec:
|
||||
metric:
|
||||
enabled: true
|
||||
rules:
|
||||
- name: dapr_runtime_service_invocation_req_sent_total
|
||||
labels:
|
||||
- name: method
|
||||
regex:
|
||||
"orders/": "orders/.+"
|
||||
```
|
||||
|
||||
When this configuration is applied, a recorded metric with the `method` label of `orders/a746dhsk293972nz` will be replaced with `orders/`.
|
||||
|
||||
### Watch the demo
|
||||
|
||||
Watch [this video to walk through handling high cardinality metrics](https://youtu.be/pOT8teL6j_k?t=1524):
|
||||
|
||||
<iframe width="560" height="315" src="https://www.youtube-nocookie.com/embed/pOT8teL6j_k?start=1524" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen></iframe>
|
||||
|
||||
|
||||
## References
|
||||
|
||||
* [Howto: Run Prometheus locally]({{< ref prometheus.md >}})
|
||||
* [Howto: Set up Prometheus and Grafana for metrics]({{< ref grafana.md >}})
|
||||
|
|
@ -0,0 +1,43 @@
|
|||
---
|
||||
type: docs
|
||||
title: "How-To: Set-up New Relic to collect and analyze metrics"
|
||||
linkTitle: "New Relic"
|
||||
weight: 6000
|
||||
description: "Set-up New Relic for Dapr metrics"
|
||||
---
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Perpetually [free New Relic account](https://newrelic.com/signup?ref=dapr), 100 GB/month of free data ingest, 1 free full access user, unlimited free basic users
|
||||
|
||||
## Background
|
||||
|
||||
New Relic offers a Prometheus OpenMetrics Integration.
|
||||
|
||||
This document explains how to install it in your cluster, either using a Helm chart (recommended).
|
||||
|
||||
## Installation
|
||||
|
||||
1. Install Helm following the official instructions.
|
||||
|
||||
2. Add the New Relic official Helm chart repository following [these instructions](https://github.com/newrelic/helm-charts/blob/master/README.md#installing-charts)
|
||||
|
||||
3. Run the following command to install the New Relic Logging Kubernetes plugin via Helm, replacing the placeholder value YOUR_LICENSE_KEY with your [New Relic license key](https://docs.newrelic.com/docs/accounts/accounts-billing/account-setup/new-relic-license-key):
|
||||
|
||||
```bash
|
||||
helm install nri-prometheus newrelic/nri-prometheus --set licenseKey=YOUR_LICENSE_KEY
|
||||
```
|
||||
|
||||
## View Metrics
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
## Related Links/References
|
||||
|
||||
* [New Relic Account Signup](https://newrelic.com/signup)
|
||||
* [Telemetry Data Platform](https://newrelic.com/platform/telemetry-data-platform)
|
||||
* [New Relic Prometheus OpenMetrics Integration](https://github.com/newrelic/helm-charts/tree/master/charts/nri-prometheus)
|
||||
* [Types of New Relic API keys](https://docs.newrelic.com/docs/apis/intro-apis/new-relic-api-keys/)
|
||||
* [Alerts and Applied Intelligence](https://docs.newrelic.com/docs/alerts-applied-intelligence/overview/)
|
||||
|
|
@ -0,0 +1,122 @@
|
|||
---
|
||||
type: docs
|
||||
title: "How-To: Observe metrics with Prometheus"
|
||||
linkTitle: "Prometheus"
|
||||
weight: 4000
|
||||
description: "Use Prometheus to collect time-series data relating to the execution of the Dapr runtime itself"
|
||||
---
|
||||
|
||||
## Setup Prometheus Locally
|
||||
To run Prometheus on your local machine, you can either [install and run it as a process](#install) or run it as a [Docker container](#Run-as-Container).
|
||||
|
||||
### Install
|
||||
{{% alert title="Note" color="warning" %}}
|
||||
You don't need to install Prometheus if you plan to run it as a Docker container. Please refer to the [Container](#run-as-container) instructions.
|
||||
{{% /alert %}}
|
||||
|
||||
To install Prometheus, follow the steps outlined [here](https://prometheus.io/docs/prometheus/latest/getting_started/) for your OS.
|
||||
|
||||
### Configure
|
||||
Now you've installed Prometheus, you need to create a configuration.
|
||||
|
||||
Below is an example Prometheus configuration, save this to a file i.e. `/tmp/prometheus.yml` or `C:\Temp\prometheus.yml`
|
||||
```yaml
|
||||
global:
|
||||
scrape_interval: 15s # By default, scrape targets every 15 seconds.
|
||||
|
||||
# A scrape configuration containing exactly one endpoint to scrape:
|
||||
# Here it's Prometheus itself.
|
||||
scrape_configs:
|
||||
- job_name: 'dapr'
|
||||
|
||||
# Override the global default and scrape targets from this job every 5 seconds.
|
||||
scrape_interval: 5s
|
||||
|
||||
static_configs:
|
||||
- targets: ['localhost:9090'] # Replace with Dapr metrics port if not default
|
||||
```
|
||||
|
||||
### Run as Process
|
||||
Run Prometheus with your configuration to start it collecting metrics from the specified targets.
|
||||
```bash
|
||||
./prometheus --config.file=/tmp/prometheus.yml --web.listen-address=:8080
|
||||
```
|
||||
> We change the port so it doesn't conflict with Dapr's own metrics endpoint.
|
||||
|
||||
If you are not currently running a Dapr application, the target will show as offline. In order to start
|
||||
collecting metrics you must start Dapr with the metrics port matching the one provided as the target in the configuration.
|
||||
|
||||
Once Prometheus is running, you'll be able to visit its dashboard by visiting `http://localhost:8080`.
|
||||
|
||||
### Run as Container
|
||||
To run Prometheus as a Docker container on your local machine, first ensure you have [Docker](https://docs.docker.com/install/) installed and running.
|
||||
|
||||
Then you can run Prometheus as a Docker container using:
|
||||
```bash
|
||||
docker run \
|
||||
--net=host \
|
||||
-v /tmp/prometheus.yml:/etc/prometheus/prometheus.yml \
|
||||
prom/prometheus --config.file=/etc/prometheus/prometheus.yml --web.listen-address=:8080
|
||||
```
|
||||
`--net=host` ensures that the Prometheus instance will be able to connect to any Dapr instances running on the host machine. If you plan to run your Dapr apps in containers as well, you'll need to run them on a shared Docker network and update the configuration with the correct target address.
|
||||
|
||||
Once Prometheus is running, you'll be able to visit its dashboard by visiting `http://localhost:8080`.
|
||||
|
||||
## Setup Prometheus on Kubernetes
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- Kubernetes (> 1.14)
|
||||
- [kubectl](https://kubernetes.io/docs/tasks/tools/)
|
||||
- [Helm 3](https://helm.sh/)
|
||||
|
||||
### Install Prometheus
|
||||
|
||||
1. First create namespace that can be used to deploy the Grafana and Prometheus monitoring tools
|
||||
|
||||
```bash
|
||||
kubectl create namespace dapr-monitoring
|
||||
```
|
||||
|
||||
2. Install Prometheus
|
||||
|
||||
```bash
|
||||
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
|
||||
helm repo update
|
||||
helm install dapr-prom prometheus-community/prometheus -n dapr-monitoring
|
||||
```
|
||||
|
||||
If you are Minikube user or want to disable persistent volume for development purposes, you can disable it by using the following command.
|
||||
|
||||
```bash
|
||||
helm install dapr-prom prometheus-community/prometheus -n dapr-monitoring
|
||||
--set alertmanager.persistence.enabled=false --set pushgateway.persistentVolume.enabled=false --set server.persistentVolume.enabled=false
|
||||
```
|
||||
|
||||
3. Validation
|
||||
|
||||
Ensure Prometheus is running in your cluster.
|
||||
|
||||
```bash
|
||||
kubectl get pods -n dapr-monitoring
|
||||
|
||||
NAME READY STATUS RESTARTS AGE
|
||||
dapr-prom-kube-state-metrics-9849d6cc6-t94p8 1/1 Running 0 4m58s
|
||||
dapr-prom-prometheus-alertmanager-749cc46f6-9b5t8 2/2 Running 0 4m58s
|
||||
dapr-prom-prometheus-node-exporter-5jh8p 1/1 Running 0 4m58s
|
||||
dapr-prom-prometheus-node-exporter-88gbg 1/1 Running 0 4m58s
|
||||
dapr-prom-prometheus-node-exporter-bjp9f 1/1 Running 0 4m58s
|
||||
dapr-prom-prometheus-pushgateway-688665d597-h4xx2 1/1 Running 0 4m58s
|
||||
dapr-prom-prometheus-server-694fd8d7c-q5d59 2/2 Running 0 4m58s
|
||||
```
|
||||
|
||||
## Example
|
||||
|
||||
<div class="embed-responsive embed-responsive-16by9">
|
||||
<iframe width="560" height="315" src="https://www.youtube-nocookie.com/embed/8W-iBDNvCUM?start=2577" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
|
||||
</div>
|
||||
|
||||
## References
|
||||
|
||||
* [Prometheus Installation](https://github.com/prometheus-community/helm-charts)
|
||||
* [Prometheus Query Language](https://prometheus.io/docs/prometheus/latest/querying/basics/)
|
||||
|
|
@ -0,0 +1,7 @@
|
|||
---
|
||||
type: docs
|
||||
title: "Tracing"
|
||||
linkTitle: "Tracing"
|
||||
weight: 200
|
||||
description: Learn about tracing scenarios and how to use tracing for visibility in your application
|
||||
---
|
||||
|
|
@ -0,0 +1,55 @@
|
|||
---
|
||||
type: docs
|
||||
title: "How-To: Set up Datadog for distributed tracing"
|
||||
linkTitle: "Datadog"
|
||||
weight: 5000
|
||||
description: "Set up Datadog for distributed tracing"
|
||||
---
|
||||
|
||||
Dapr captures metrics and traces that can be sent directly to Datadog through the OpenTelemetry Collector Datadog exporter.
|
||||
|
||||
## Configure Dapr tracing with the OpenTelemetry Collector and Datadog
|
||||
|
||||
Using the OpenTelemetry Collector Datadog exporter, you can configure Dapr to create traces for each application in your Kubernetes cluster and collect them in Datadog.
|
||||
|
||||
> Before you begin, [set up the OpenTelemetry Collector]({{< ref "open-telemetry-collector.md#setting-opentelemetry-collector" >}}).
|
||||
|
||||
1. Add your Datadog API key to the `./deploy/opentelemetry-collector-generic-datadog.yaml` file in the `datadog` exporter configuration section:
|
||||
```yaml
|
||||
data:
|
||||
otel-collector-config:
|
||||
...
|
||||
exporters:
|
||||
...
|
||||
datadog:
|
||||
api:
|
||||
key: <YOUR_API_KEY>
|
||||
```
|
||||
|
||||
1. Apply the `opentelemetry-collector` configuration by running the following command.
|
||||
|
||||
```sh
|
||||
kubectl apply -f ./deploy/open-telemetry-collector-generic-datadog.yaml
|
||||
```
|
||||
|
||||
1. Set up a Dapr configuration file that will turn on tracing and deploy a tracing exporter component that uses the OpenTelemetry Collector.
|
||||
|
||||
```sh
|
||||
kubectl apply -f ./deploy/collector-config.yaml
|
||||
|
||||
1. Apply the `appconfig` configuration by adding a `dapr.io/config` annotation to the container that you want to participate in the distributed tracing.
|
||||
|
||||
```yml
|
||||
annotations:
|
||||
dapr.io/config: "appconfig"
|
||||
|
||||
1. Create and configure the application. Once running, telemetry data is sent to Datadog and visible in Datadog APM.
|
||||
|
||||
<img src="/images/datadog-traces.png" width=1200 alt="Datadog APM showing telemetry data.">
|
||||
|
||||
|
||||
## Related Links/References
|
||||
|
||||
* [Complete example of setting up Dapr on a Kubernetes cluster](https://github.com/ericmustin/quickstarts/tree/master/hello-kubernetes)
|
||||
* [Datadog documentation about OpenTelemetry support](https://docs.datadoghq.com/opentelemetry/)
|
||||
* [Datadog Application Performance Monitoring](https://docs.datadoghq.com/tracing/)
|
||||
|
|
@ -0,0 +1,187 @@
|
|||
---
|
||||
type: docs
|
||||
title: "How-To: Set up Jaeger for distributed tracing"
|
||||
linkTitle: "Jaeger"
|
||||
weight: 3000
|
||||
description: "Set up Jaeger for distributed tracing"
|
||||
type: docs
|
||||
---
|
||||
|
||||
Dapr supports the Zipkin protocol. Since Jaeger is compatible with Zipkin, the Zipkin protocol can be used to communication with Jaeger.
|
||||
|
||||
## Configure self hosted mode
|
||||
|
||||
### Setup
|
||||
|
||||
The simplest way to start Jaeger is to use the pre-built all-in-one Jaeger image published to DockerHub:
|
||||
|
||||
```bash
|
||||
docker run -d --name jaeger \
|
||||
-e COLLECTOR_ZIPKIN_HOST_PORT=:9412 \
|
||||
-p 16686:16686 \
|
||||
-p 9412:9412 \
|
||||
jaegertracing/all-in-one:1.22
|
||||
```
|
||||
|
||||
|
||||
Next, create the following YAML files locally:
|
||||
|
||||
* **config.yaml**: Note that because we are using the Zipkin protocol
|
||||
to talk to Jaeger, we specify the `zipkin` section of tracing
|
||||
configuration set the `endpointAddress` to address of the Jaeger
|
||||
instance.
|
||||
|
||||
```yaml
|
||||
apiVersion: dapr.io/v1alpha1
|
||||
kind: Configuration
|
||||
metadata:
|
||||
name: tracing
|
||||
namespace: default
|
||||
spec:
|
||||
tracing:
|
||||
samplingRate: "1"
|
||||
zipkin:
|
||||
endpointAddress: "http://localhost:9412/api/v2/spans"
|
||||
```
|
||||
|
||||
To launch the application referring to the new YAML file, you can use
|
||||
`--config` option:
|
||||
|
||||
```bash
|
||||
dapr run --app-id mynode --app-port 3000 node app.js --config config.yaml
|
||||
```
|
||||
|
||||
### Viewing Traces
|
||||
To view traces, in your browser go to http://localhost:16686 to see the Jaeger UI.
|
||||
|
||||
## Configure Kubernetes
|
||||
The following steps shows you how to configure Dapr to send distributed tracing data to Jaeger running as a container in your Kubernetes cluster, how to view them.
|
||||
|
||||
### Setup
|
||||
|
||||
First create the following YAML file to install Jaeger, file name is `jaeger-operator.yaml`
|
||||
|
||||
#### Development and test
|
||||
|
||||
By default, the allInOne Jaeger image uses memory as the backend storage and it is not recommended to use this in a production environment.
|
||||
|
||||
```yaml
|
||||
apiVersion: jaegertracing.io/v1
|
||||
kind: "Jaeger"
|
||||
metadata:
|
||||
name: jaeger
|
||||
spec:
|
||||
strategy: allInOne
|
||||
ingress:
|
||||
enabled: false
|
||||
allInOne:
|
||||
image: jaegertracing/all-in-one:1.22
|
||||
options:
|
||||
query:
|
||||
base-path: /jaeger
|
||||
```
|
||||
|
||||
#### Production
|
||||
Jaeger uses Elasticsearch as the backend storage, and you can create a secret in k8s cluster to access Elasticsearch server with access control.
|
||||
|
||||
|
||||
```shell
|
||||
kubectl create secret generic jaeger-secret --from-literal=ES_PASSWORD='xxx' --from-literal=ES_USERNAME='xxx' -n ${NAMESPACE}
|
||||
```
|
||||
|
||||
```yaml
|
||||
apiVersion: jaegertracing.io/v1
|
||||
kind: "Jaeger"
|
||||
metadata:
|
||||
name: jaeger
|
||||
spec:
|
||||
strategy: production
|
||||
query:
|
||||
options:
|
||||
log-level: info
|
||||
query:
|
||||
base-path: /jaeger
|
||||
collector:
|
||||
maxReplicas: 5
|
||||
resources:
|
||||
limits:
|
||||
cpu: 500m
|
||||
memory: 516Mi
|
||||
storage:
|
||||
type: elasticsearch
|
||||
esIndexCleaner:
|
||||
enabled: false ## turn the job deployment on and off
|
||||
numberOfDays: 7 ## number of days to wait before deleting a record
|
||||
schedule: "55 23 * * *" ## cron expression for it to run
|
||||
image: jaegertracing/jaeger-es-index-cleaner ## image of the job
|
||||
secretName: jaeger-secret
|
||||
options:
|
||||
es:
|
||||
server-urls: http://elasticsearch:9200
|
||||
```
|
||||
|
||||
The pictures are as follows, include Elasticsearch and Grafana tracing data:
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
Now, use the above YAML file to install Jaeger
|
||||
|
||||
```bash
|
||||
# Install Jaeger
|
||||
helm repo add jaegertracing https://jaegertracing.github.io/helm-charts
|
||||
helm install jaeger-operator jaegertracing/jaeger-operator
|
||||
kubectl apply -f jaeger-operator.yaml
|
||||
|
||||
# Wait for Jaeger to be up and running
|
||||
kubectl wait deploy --selector app.kubernetes.io/name=jaeger --for=condition=available
|
||||
```
|
||||
|
||||
Next, create the following YAML file locally:
|
||||
|
||||
* **tracing.yaml**
|
||||
|
||||
```yaml
|
||||
apiVersion: dapr.io/v1alpha1
|
||||
kind: Configuration
|
||||
metadata:
|
||||
name: tracing
|
||||
namespace: default
|
||||
spec:
|
||||
tracing:
|
||||
samplingRate: "1"
|
||||
zipkin:
|
||||
endpointAddress: "http://jaeger-collector.default.svc.cluster.local:9411/api/v2/spans"
|
||||
```
|
||||
|
||||
Finally, deploy the the Dapr component and configuration files:
|
||||
|
||||
```bash
|
||||
kubectl apply -f tracing.yaml
|
||||
```
|
||||
|
||||
In order to enable this configuration for your Dapr sidecar, add the following annotation to your pod spec template:
|
||||
|
||||
```yml
|
||||
annotations:
|
||||
dapr.io/config: "tracing"
|
||||
```
|
||||
|
||||
That's it! Your Dapr sidecar is now configured for use with Jaeger.
|
||||
|
||||
### Viewing Tracing Data
|
||||
|
||||
To view traces, connect to the Jaeger Service and open the UI:
|
||||
|
||||
```bash
|
||||
kubectl port-forward svc/jaeger-query 16686
|
||||
```
|
||||
|
||||
In your browser, go to `http://localhost:16686` and you will see the Jaeger UI.
|
||||
|
||||
|
||||
|
||||
## References
|
||||
- [Jaeger Getting Started](https://www.jaegertracing.io/docs/1.21/getting-started/#all-in-one)
|
||||
|
|
@ -0,0 +1,114 @@
|
|||
---
|
||||
type: docs
|
||||
title: "How-To: Set-up New Relic for distributed tracing"
|
||||
linkTitle: "New Relic"
|
||||
weight: 2000
|
||||
description: "Set-up New Relic for distributed tracing"
|
||||
---
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Perpetually [free New Relic account](https://newrelic.com/signup?ref=dapr), 100 GB/month of free data ingest, 1 free full access user, unlimited free basic users
|
||||
|
||||
## Configure Dapr tracing
|
||||
|
||||
Dapr natively captures metrics and traces that can be send directly to New Relic. The easiest way to export these is by configuring Dapr to send the traces to [New Relic's Trace API](https://docs.newrelic.com/docs/distributed-tracing/trace-api/report-zipkin-format-traces-trace-api/) using the Zipkin trace format.
|
||||
|
||||
In order for the integration to send data to New Relic [Telemetry Data Platform](https://newrelic.com/platform/telemetry-data-platform), you need a [New Relic Insights Insert API key](https://docs.newrelic.com/docs/apis/intro-apis/new-relic-api-keys/#insights-insert-key).
|
||||
|
||||
```yaml
|
||||
apiVersion: dapr.io/v1alpha1
|
||||
kind: Configuration
|
||||
metadata:
|
||||
name: appconfig
|
||||
namespace: default
|
||||
spec:
|
||||
tracing:
|
||||
samplingRate: "1"
|
||||
zipkin:
|
||||
endpointAddress: "https://trace-api.newrelic.com/trace/v1?Api-Key=<NR-INSIGHTS-INSERT-API-KEY>&Data-Format=zipkin&Data-Format-Version=2"
|
||||
```
|
||||
|
||||
### Viewing Traces
|
||||
|
||||
New Relic Distributed Tracing overview
|
||||
|
||||
|
||||
New Relic Distributed Tracing details
|
||||
|
||||
|
||||
## (optional) New Relic Instrumentation
|
||||
|
||||
In order for the integrations to send data to New Relic Telemetry Data Platform, you either need a [New Relic license key](https://docs.newrelic.com/docs/accounts/accounts-billing/account-setup/new-relic-license-key) or [New Relic Insights Insert API key](https://docs.newrelic.com/docs/apis/intro-apis/new-relic-api-keys/#insights-insert-key).
|
||||
|
||||
### OpenTelemetry instrumentation
|
||||
|
||||
Leverage the different language specific OpenTelemetry implementations, for example [New Relic Telemetry SDK and OpenTelemetry support for .NET](https://github.com/newrelic/newrelic-telemetry-sdk-dotnet). In this case, use the [OpenTelemetry Trace Exporter](https://github.com/newrelic/newrelic-telemetry-sdk-dotnet/tree/main/src/NewRelic.OpenTelemetry). See example [here](https://github.com/harrykimpel/quickstarts/blob/master/distributed-calculator/csharp-otel/Startup.cs).
|
||||
|
||||
### New Relic Language agent
|
||||
|
||||
Similarly to the OpenTelemetry instrumentation, you can also leverage a New Relic language agent. As an example, the [New Relic agent instrumentation for .NET Core](https://docs.newrelic.com/docs/agents/net-agent/other-installation/install-net-agent-docker-container) is part of the Dockerfile. See example [here](https://github.com/harrykimpel/quickstarts/blob/master/distributed-calculator/csharp/Dockerfile).
|
||||
|
||||
## (optional) Enable New Relic Kubernetes integration
|
||||
|
||||
In case Dapr and your applications run in the context of a Kubernetes environment, you can enable additional metrics and logs.
|
||||
|
||||
The easiest way to install the New Relic Kubernetes integration is to use the [automated installer](https://one.newrelic.com/launcher/nr1-core.settings?pane=eyJuZXJkbGV0SWQiOiJrOHMtY2x1c3Rlci1leHBsb3Jlci1uZXJkbGV0Lms4cy1zZXR1cCJ9) to generate a manifest. It bundles not just the integration DaemonSets, but also other New Relic Kubernetes configurations, like [Kubernetes events](https://docs.newrelic.com/docs/integrations/kubernetes-integration/kubernetes-events/install-kubernetes-events-integration), [Prometheus OpenMetrics](https://docs.newrelic.com/docs/integrations/prometheus-integrations/get-started/send-prometheus-metric-data-new-relic/), and [New Relic log monitoring](https://docs.newrelic.com/docs/logs/ui-data/use-logs-ui/).
|
||||
|
||||
### New Relic Kubernetes Cluster Explorer
|
||||
|
||||
The [New Relic Kubernetes Cluster Explorer](https://docs.newrelic.com/docs/integrations/kubernetes-integration/understand-use-data/kubernetes-cluster-explorer) provides a unique visualization of the entire data and deployments of the data collected by the Kubernetes integration.
|
||||
|
||||
It is a good starting point to observe all your data and dig deeper into any performance issues or incidents happening inside of the application or microservices.
|
||||
|
||||
|
||||
|
||||
Automated correlation is part of the visualization capabilities of New Relic.
|
||||
|
||||
### Pod-level details
|
||||
|
||||
|
||||
|
||||
### Logs in Context
|
||||
|
||||
|
||||
|
||||
## New Relic Dashboards
|
||||
|
||||
### Kubernetes Overview
|
||||
|
||||
|
||||
|
||||
### Dapr System Services
|
||||
|
||||
|
||||
|
||||
### Dapr Metrics
|
||||
|
||||
|
||||
|
||||
## New Relic Grafana integration
|
||||
|
||||
New Relic teamed up with [Grafana Labs](https://grafana.com/) so you can use the [Telemetry Data Platform](https://newrelic.com/platform/telemetry-data-platform) as a data source for Prometheus metrics and see them in your existing dashboards, seamlessly tapping into the reliability, scale, and security provided by New Relic.
|
||||
|
||||
[Grafana dashboard templates](https://github.com/dapr/dapr/blob/227028e7b76b7256618cd3236d70c1d4a4392c9a/grafana/README.md) to monitor Dapr system services and sidecars can easily be used without any changes. New Relic provides a [native endpoint for Prometheus metrics](https://docs.newrelic.com/docs/integrations/grafana-integrations/set-configure/configure-new-relic-prometheus-data-source-grafana) into Grafana. A datasource can easily be set-up:
|
||||
|
||||
|
||||
|
||||
And the exact same dashboard templates from Dapr can be imported to visualize Dapr system services and sidecars.
|
||||
|
||||
|
||||
|
||||
## New Relic Alerts
|
||||
|
||||
All the data that is collected from Dapr, Kubernetes or any services that run on top of can be used to set-up alerts and notifications into the preferred channel of your choice. See [Alerts and Applied Intelligence](https://docs.newrelic.com/docs/alerts-applied-intelligence/overview/).
|
||||
|
||||
## Related Links/References
|
||||
|
||||
* [New Relic Account Signup](https://newrelic.com/signup)
|
||||
* [Telemetry Data Platform](https://newrelic.com/platform/telemetry-data-platform)
|
||||
* [Distributed Tracing](https://docs.newrelic.com/docs/distributed-tracing/concepts/introduction-distributed-tracing/)
|
||||
* [New Relic Trace API](https://docs.newrelic.com/docs/distributed-tracing/trace-api/introduction-trace-api/)
|
||||
* [Types of New Relic API keys](https://docs.newrelic.com/docs/apis/intro-apis/new-relic-api-keys/)
|
||||
* [New Relic OpenTelemetry User Experience](https://blog.newrelic.com/product-news/opentelemetry-user-experience/)
|
||||
* [Alerts and Applied Intelligence](https://docs.newrelic.com/docs/alerts-applied-intelligence/overview/)
|
||||
|
|
@ -0,0 +1,7 @@
|
|||
---
|
||||
type: docs
|
||||
title: "Open Telemetry Collector"
|
||||
linkTitle: "Open Telemetry Collector"
|
||||
weight: 700
|
||||
description: "How to set up your observability tools to receive application traces"
|
||||
---
|
||||
|
|
@ -0,0 +1,72 @@
|
|||
---
|
||||
type: docs
|
||||
title: "Using OpenTelemetry Collector to collect traces to send to AppInsights"
|
||||
linkTitle: "Using the OpenTelemetry for Azure AppInsights"
|
||||
weight: 1000
|
||||
description: "How to push trace events to Azure Application Insights, using the OpenTelemetry Collector."
|
||||
---
|
||||
|
||||
Dapr integrates with [OpenTelemetry Collector](https://github.com/open-telemetry/opentelemetry-collector) using the Zipkin API. This guide walks through an example using Dapr to push trace events to Azure Application Insights, using the OpenTelemetry Collector.
|
||||
|
||||
## Requirements
|
||||
|
||||
A installation of Dapr on Kubernetes.
|
||||
|
||||
## How to configure distributed tracing with Application Insights
|
||||
|
||||
### Setup Application Insights
|
||||
|
||||
1. First, you'll need an Azure account. See instructions [here](https://azure.microsoft.com/free/) to apply for a **free** Azure account.
|
||||
2. Follow instructions [here](https://docs.microsoft.com/azure/azure-monitor/app/create-new-resource) to create a new Application Insights resource.
|
||||
3. Get the Application Insights Intrumentation key from your Application Insights page.
|
||||
|
||||
### Run OpenTelemetry Collector to push to your Application Insights instance
|
||||
|
||||
Install the OpenTelemetry Collector to your Kubernetes cluster to push events to your Application Insights instance
|
||||
|
||||
1. Check out the file [open-telemetry-collector-appinsights.yaml](/docs/open-telemetry-collector/open-telemetry-collector-appinsights.yaml) and replace the `<INSTRUMENTATION-KEY>` placeholder with your Application Insights Instrumentation Key.
|
||||
|
||||
2. Apply the configuration with `kubectl apply -f open-telemetry-collector-appinsights.yaml`.
|
||||
|
||||
Next, set up both a Dapr configuration file to turn on tracing and deploy a tracing exporter component that uses the OpenTelemetry Collector.
|
||||
|
||||
1. Create a collector-config.yaml file with this [content](/docs/open-telemetry-collector/collector-config.yaml)
|
||||
|
||||
2. Apply the configuration with `kubectl apply -f collector-config.yaml`.
|
||||
|
||||
### Deploy your app with tracing
|
||||
|
||||
When running in Kubernetes mode, apply the `appconfig` configuration by adding a `dapr.io/config` annotation to the container that you want to participate in the distributed tracing, as shown in the following example:
|
||||
|
||||
```yaml
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
...
|
||||
spec:
|
||||
...
|
||||
template:
|
||||
metadata:
|
||||
...
|
||||
annotations:
|
||||
dapr.io/enabled: "true"
|
||||
dapr.io/app-id: "MyApp"
|
||||
dapr.io/app-port: "8080"
|
||||
dapr.io/config: "appconfig"
|
||||
```
|
||||
|
||||
Some of the quickstarts such as [distributed calculator](https://github.com/dapr/quickstarts/tree/master/tutorials/distributed-calculator) already configure these settings, so if you are using those no additional settings are needed.
|
||||
|
||||
That's it! There's no need include any SDKs or instrument your application code. Dapr automatically handles the distributed tracing for you.
|
||||
|
||||
> **NOTE**: You can register multiple tracing exporters at the same time, and the tracing logs are forwarded to all registered exporters.
|
||||
|
||||
Deploy and run some applications. After a few minutes, you should see tracing logs appearing in your Application Insights resource. You can also use the **Application Map** to examine the topology of your services, as shown below:
|
||||
|
||||
|
||||
|
||||
> **NOTE**: Only operations going through Dapr API exposed by Dapr sidecar (e.g. service invocation or event publishing) are displayed in Application Map topology.
|
||||
|
||||
## Related links
|
||||
* Try out the [observability quickstart](https://github.com/dapr/quickstarts/tree/master/tutorials/observability/README.md)
|
||||
* How to set [tracing configuration options]({{< ref "configuration-overview.md#tracing" >}})
|
||||
|
|
@ -0,0 +1,74 @@
|
|||
---
|
||||
type: docs
|
||||
title: "Using OpenTelemetry Collector to collect traces"
|
||||
linkTitle: "Using the OpenTelemetry Collector"
|
||||
weight: 900
|
||||
description: "How to use Dapr to push trace events through the OpenTelemetry Collector."
|
||||
---
|
||||
|
||||
{{% alert title="Note" color="primary" %}}
|
||||
Dapr directly writes traces using the OpenTelemetry (OTEL) protocol as the recommended method. For observability tools that support OTEL protocol, you do not need to use the OpenTelemetry Collector.
|
||||
|
||||
Dapr can also write traces using the Zipkin protocol. Previous to supporting the OTEL protocol, combining the Zipkin protocol with the [OpenTelemetry Collector](https://github.com/open-telemetry/opentelemetry-collector) enabled you to send traces to observability tools such as AWS X-Ray, Google Cloud Operations Suite, and Azure AppInsights. This approach remains for reference purposes only.
|
||||
{{% /alert %}}
|
||||
|
||||
|
||||
|
||||
## Requirements
|
||||
|
||||
1. A installation of Dapr on Kubernetes.
|
||||
|
||||
2. You are already setting up your trace backends to receive traces.
|
||||
|
||||
3. Check OpenTelemetry Collector exporters [here](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter) and [here](https://github.com/open-telemetry/opentelemetry-collector/tree/main/exporter) to see if your trace backend is supported by the OpenTelemetry Collector. On those linked pages, find the exporter you want to use and read its doc to find out the parameters required.
|
||||
|
||||
## Setting OpenTelemetry Collector
|
||||
|
||||
### Run OpenTelemetry Collector to push to your trace backend
|
||||
|
||||
1. Check out the file [open-telemetry-collector-generic.yaml](/docs/open-telemetry-collector/open-telemetry-collector-generic.yaml) and replace the section marked with `<your-exporter-here>` with the correct settings for your trace exporter. Again, refer to the OpenTelemetry Collector links in the Prerequisites section to determine the correct settings.
|
||||
|
||||
2. Apply the configuration with `kubectl apply -f open-telemetry-collector-generic.yaml`.
|
||||
|
||||
## Set up Dapr to send trace to OpenTelemetry Collector
|
||||
|
||||
### Turn on tracing in Dapr
|
||||
Next, set up both a Dapr configuration file to turn on tracing and deploy a tracing exporter component that uses the OpenTelemetry Collector.
|
||||
|
||||
1. Create a collector-config.yaml file with this [content](/docs/open-telemetry-collector/collector-config.yaml)
|
||||
|
||||
2. Apply the configuration with `kubectl apply -f collector-config.yaml`.
|
||||
|
||||
### Deploy your app with tracing
|
||||
|
||||
When running in Kubernetes mode, apply the `appconfig` configuration by adding a `dapr.io/config` annotation to the container that you want to participate in the distributed tracing, as shown in the following example:
|
||||
|
||||
```yaml
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
...
|
||||
spec:
|
||||
...
|
||||
template:
|
||||
metadata:
|
||||
...
|
||||
annotations:
|
||||
dapr.io/enabled: "true"
|
||||
dapr.io/app-id: "MyApp"
|
||||
dapr.io/app-port: "8080"
|
||||
dapr.io/config: "appconfig"
|
||||
```
|
||||
|
||||
Some of the quickstarts such as [distributed calculator](https://github.com/dapr/quickstarts/tree/master/tutorials/distributed-calculator) already configure these settings, so if you are using those no additional settings are needed.
|
||||
|
||||
That's it! There's no need include any SDKs or instrument your application code. Dapr automatically handles the distributed tracing for you.
|
||||
|
||||
> **NOTE**: You can register multiple tracing exporters at the same time, and the tracing logs are forwarded to all registered exporters.
|
||||
|
||||
Deploy and run some applications. Wait for the trace to propagate to your tracing backend and view them there.
|
||||
|
||||
## Related links
|
||||
* Try out the [observability quickstart](https://github.com/dapr/quickstarts/tree/master/tutorials/observability/README.md)
|
||||
* How to set [tracing configuration options]({{< ref "configuration-overview.md#tracing" >}})
|
||||
|
||||
|
|
@ -0,0 +1,80 @@
|
|||
---
|
||||
type: docs
|
||||
title: "Configure Dapr to send distributed tracing data"
|
||||
linkTitle: "Configure tracing"
|
||||
weight: 30
|
||||
description: "Set up Dapr to send distributed tracing data"
|
||||
---
|
||||
|
||||
{{% alert title="Note" color="primary" %}}
|
||||
It is recommended to run Dapr with tracing enabled for any production scenario. You can configure Dapr to send tracing and telemetry data to many observability tools based on your environment, whether it is running in the cloud or on-premises.
|
||||
{{% /alert %}}
|
||||
|
||||
|
||||
## Configuration
|
||||
|
||||
The `tracing` section under the `Configuration` spec contains the following properties:
|
||||
|
||||
```yml
|
||||
spec:
|
||||
tracing:
|
||||
samplingRate: "1"
|
||||
otel:
|
||||
endpointAddress: "https://..."
|
||||
zipkin:
|
||||
endpointAddress: "https://..."
|
||||
|
||||
```
|
||||
|
||||
The following table lists the properties for tracing:
|
||||
|
||||
| Property | Type | Description |
|
||||
|--------------|--------|-------------|
|
||||
| `samplingRate` | string | Set sampling rate for tracing to be enabled or disabled.
|
||||
| `stdout` | bool | True write more verbose information to the traces
|
||||
| `otel.endpointAddress` | string | Set the Open Telemetry (OTEL) server address.
|
||||
| `otel.isSecure` | bool | Is the connection to the endpoint address encrypted.
|
||||
| `otel.protocol` | string | Set to `http` or `grpc` protocol.
|
||||
| `zipkin.endpointAddress` | string | Set the Zipkin server address. If this is used, you do not need to specify the `otel` section.
|
||||
|
||||
To enable tracing, use a configuration file (in self hosted mode) or a Kubernetes configuration object (in Kubernetes mode). For example, the following configuration object changes the sample rate to 1 (every span is sampled), and sends trace using OTEL protocol to the OTEL server at localhost:4317
|
||||
|
||||
```yaml
|
||||
apiVersion: dapr.io/v1alpha1
|
||||
kind: Configuration
|
||||
metadata:
|
||||
name: tracing
|
||||
spec:
|
||||
tracing:
|
||||
samplingRate: "1"
|
||||
otel:
|
||||
endpointAddress: "localhost:4317"
|
||||
isSecure: false
|
||||
protocol: grpc
|
||||
```
|
||||
|
||||
## Sampling rate
|
||||
|
||||
Dapr uses probabilistic sampling. The sample rate defines the probability a tracing span will be sampled and can have a value between 0 and 1 (inclusive). The default sample rate is 0.0001 (i.e. 1 in 10,000 spans is sampled).
|
||||
|
||||
Changing `samplingRate` to 0 disables tracing altogether.
|
||||
|
||||
## Environment variables
|
||||
|
||||
The OpenTelemetry (otel) endpoint can also be configured via an environment variables. The presence of the OTEL_EXPORTER_OTLP_ENDPOINT environment variable
|
||||
turns on tracing for the sidecar.
|
||||
|
||||
| Environment Variable | Description |
|
||||
|----------------------|-------------|
|
||||
| `OTEL_EXPORTER_OTLP_ENDPOINT` | Sets the Open Telemetry (OTEL) server address, turns on tracing |
|
||||
| `OTEL_EXPORTER_OTLP_INSECURE` | Sets the connection to the endpoint as unencrypted (true/false) |
|
||||
| `OTEL_EXPORTER_OTLP_PROTOCOL` | Transport protocol (`grpc`, `http/protobuf`, `http/json`) |
|
||||
|
||||
## Next steps
|
||||
|
||||
Learn how to set up tracing with one of the following tools:
|
||||
- [OTEL Collector]({{< ref otel-collector >}})
|
||||
- [New Relic]({{< ref newrelic.md >}})
|
||||
- [Jaeger]({{< ref jaeger.md >}})
|
||||
- [Zipkin]({{< ref zipkin.md >}})
|
||||
- [Datadog]({{< ref datadog.md >}})
|
||||
|
|
@ -0,0 +1,113 @@
|
|||
---
|
||||
type: docs
|
||||
title: "Distributed tracing overview"
|
||||
linkTitle: "Overview"
|
||||
weight: 10
|
||||
description: "Overview on using tracing to get visibility into your application"
|
||||
---
|
||||
|
||||
Dapr uses the Open Telemetry (OTEL) and Zipkin protocols for distributed traces. OTEL is the industry standard and is the recommended trace protocol to use.
|
||||
|
||||
Most observability tools support OTEL, including:
|
||||
- [Google Cloud Operations](https://cloud.google.com/products/operations)
|
||||
- [New Relic](https://newrelic.com)
|
||||
- [Azure Monitor](https://azure.microsoft.com/services/monitor/)
|
||||
- [Datadog](https://www.datadoghq.com)
|
||||
- Instana
|
||||
- [Jaeger](https://www.jaegertracing.io/)
|
||||
- [SignalFX](https://www.signalfx.com/)
|
||||
|
||||
## Scenarios
|
||||
|
||||
Tracing is used with service invocaton and pub/sub APIs. You can flow trace context between services that uses these APIs. There are two scenarios for how tracing is used:
|
||||
|
||||
1. Dapr generates the trace context and you propagate the trace context to another service.
|
||||
1. You generate the trace context and Dapr propagates the trace context to a service.
|
||||
|
||||
### Scenario 1: Dapr generates trace context headers
|
||||
|
||||
#### Propagating sequential service calls
|
||||
|
||||
Dapr takes care of creating the trace headers. However, when there are more than two services, you're responsible for propagating the trace headers between them. Let's go through the scenarios with examples:
|
||||
|
||||
##### Single service invocation call
|
||||
|
||||
For example, `service A -> service B`.
|
||||
|
||||
Dapr generates the trace headers in `service A`, which are then propagated from `service A` to `service B`. No further propagation is needed.
|
||||
|
||||
##### Multiple sequential service invocation calls
|
||||
|
||||
For example, `service A -> service B -> propagate trace headers to -> service C` and so on to further Dapr-enabled services.
|
||||
|
||||
Dapr generates the trace headers at the beginning of the request in `service A`, which are then propagated to `service B`. You are now responsible for taking the headers and propagating them to `service C`, since this is specific to your application.
|
||||
|
||||
In other words, if the app is calling to Dapr and wants to trace with an existing trace header (span), it must always propagate to Dapr (from `service B` to `service C`, in this example). Dapr always propagates trace spans to an application.
|
||||
|
||||
{{% alert title="Note" color="primary" %}}
|
||||
No helper methods are exposed in Dapr SDKs to propagate and retrieve trace context. You need to use HTTP/gRPC clients to propagate and retrieve trace headers through HTTP headers and gRPC metadata.
|
||||
{{% /alert %}}
|
||||
|
||||
##### Request is from external endpoint
|
||||
|
||||
For example, `from a gateway service to a Dapr-enabled service A`.
|
||||
|
||||
An external gateway ingress calls Dapr, which generates the trace headers and calls `service A`. `Service A` then calls `service B` and further Dapr-enabled services.
|
||||
|
||||
You must propagate the headers from `service A` to `service B`. For example: `Ingress -> service A -> propagate trace headers -> service B`. This is similar to [case 2]({{< ref "tracing-overview.md#multiple-sequential-service-invocation-calls" >}}).
|
||||
|
||||
##### Pub/sub messages
|
||||
|
||||
Dapr generates the trace headers in the published message topic. These trace headers are propagated to any services listening on that topic.
|
||||
|
||||
#### Propagating multiple different service calls
|
||||
|
||||
In the following scenarios, Dapr does some of the work for you, with you then creating or propagating trace headers.
|
||||
|
||||
##### Multiple service calls to different services from single service
|
||||
|
||||
When you are calling multiple services from a single service, you need to propagate the trace headers. For example:
|
||||
|
||||
```
|
||||
service A -> service B
|
||||
[ .. some code logic ..]
|
||||
service A -> service C
|
||||
[ .. some code logic ..]
|
||||
service A -> service D
|
||||
[ .. some code logic ..]
|
||||
```
|
||||
|
||||
In this case:
|
||||
1. When `service A` first calls `service B`, Dapr generates the trace headers in `service A`.
|
||||
1. The trace headers in `service A` are propagated to `service B`.
|
||||
1. These trace headers are returned in the response from `service B` as part of response headers.
|
||||
1. You then need to propagate the returned trace context to the next services, like `service C` and `service D`, as Dapr does not know you want to reuse the same header.
|
||||
|
||||
### Scenario 2: You generate your own trace context headers from non-Daprized applications
|
||||
|
||||
Generating your own trace context headers is more unusual and typically not required when calling Dapr.
|
||||
|
||||
However, there are scenarios where you could specifically choose to add W3C trace headers into a service call. For example, you have an existing application that does not use Dapr. In this case, Dapr still propagates the trace context headers for you.
|
||||
|
||||
If you decide to generate trace headers yourself, there are three ways this can be done:
|
||||
|
||||
1. Standard OpenTelemetry SDK
|
||||
|
||||
You can use the industry standard [OpenTelemetry SDKs](https://opentelemetry.io/docs/instrumentation/) to generate trace headers and pass these trace headers to a Dapr-enabled service. _This is the preferred method_.
|
||||
|
||||
1. Vendor SDK
|
||||
|
||||
You can use a vendor SDK that provides a way to generate W3C trace headers and pass them to a Dapr-enabled service.
|
||||
|
||||
1. W3C trace context
|
||||
|
||||
You can handcraft a trace context following [W3C trace context specifications](https://www.w3.org/TR/trace-context/) and pass them to a Dapr-enabled service.
|
||||
|
||||
Read [the trace context overview]({{< ref w3c-tracing-overview >}}) for more background and examples on W3C trace context and headers.
|
||||
|
||||
## Related Links
|
||||
|
||||
- [Observability concepts]({{< ref observability-concept.md >}})
|
||||
- [W3C Trace Context for distributed tracing]({{< ref w3c-tracing-overview >}})
|
||||
- [W3C Trace Context specification](https://www.w3.org/TR/trace-context/)
|
||||
- [Observability quickstart](https://github.com/dapr/quickstarts/tree/master/tutorials/observability)
|
||||
|
|
@ -0,0 +1,90 @@
|
|||
---
|
||||
type: docs
|
||||
title: "W3C trace context overview"
|
||||
linkTitle: "W3C trace context"
|
||||
weight: 20
|
||||
description: Background and scenarios for using W3C tracing context and headers with Dapr
|
||||
---
|
||||
|
||||
Dapr uses the [Open Telemetry protocol](https://opentelemetry.io/), which in turn uses the [W3C trace context](https://www.w3.org/TR/trace-context/) for distributed tracing for both service invocation and pub/sub messaging. Dapr generates and propagates the trace context information, which can be sent to observability tools for visualization and querying.
|
||||
|
||||
## Background
|
||||
|
||||
Distributed tracing is a methodology implemented by tracing tools to follow, analyze, and debug a transaction across multiple software components.
|
||||
|
||||
Typically, a distributed trace traverses more than one service, which requires it to be uniquely identifiable. **Trace context propagation** passes along this unique identification.
|
||||
|
||||
In the past, trace context propagation was implemented individually by each different tracing vendor. In multi-vendor environments, this causes interoperability problems, such as:
|
||||
|
||||
- Traces collected by different tracing vendors can't be correlated, as there is no shared unique identifier.
|
||||
- Traces crossing boundaries between different tracing vendors can't be propagated, as there is no forwarded, uniformly agreed set of identification.
|
||||
- Vendor-specific metadata might be dropped by intermediaries.
|
||||
- Cloud platform vendors, intermediaries, and service providers cannot guarantee to support trace context propagation, as there is no standard to follow.
|
||||
|
||||
Previously, most applications were monitored by a single tracing vendor and stayed within the boundaries of a single platform provider, so these problems didn't have a significant impact.
|
||||
|
||||
Today, an increasing number of applications are distributed and leverage multiple middleware services and cloud platforms. This transformation of modern applications requires a distributed tracing context propagation standard.
|
||||
|
||||
The [W3C trace context specification](https://www.w3.org/TR/trace-context/) defines a universally agreed-upon format for the exchange of trace context propagation data (referred to as trace context). Trace context solves the above problems by providing:
|
||||
|
||||
- A unique identifier for individual traces and requests, allowing trace data of multiple providers to be linked together.
|
||||
- An agreed-upon mechanism to forward vendor-specific trace data and avoid broken traces when multiple tracing tools participate in a single transaction.
|
||||
- An industry standard that intermediaries, platforms, and hardware providers can support.
|
||||
|
||||
This unified approach for propagating trace data improves visibility into the behavior of distributed applications, facilitating problem and performance analysis.
|
||||
|
||||
## W3C trace context and headers format
|
||||
|
||||
### W3C trace context
|
||||
|
||||
Dapr uses the standard W3C trace context headers.
|
||||
|
||||
- For HTTP requests, Dapr uses `traceparent` header.
|
||||
- For gRPC requests, Dapr uses `grpc-trace-bin` header.
|
||||
|
||||
When a request arrives without a trace ID, Dapr creates a new one. Otherwise, it passes the trace ID along the call chain.
|
||||
|
||||
### W3C trace headers
|
||||
These are the specific trace context headers that are generated and propagated by Dapr for HTTP and gRPC.
|
||||
|
||||
{{< tabs "HTTP" "gRPC" >}}
|
||||
<!-- HTTP -->
|
||||
{{% codetab %}}
|
||||
|
||||
Copy these headers when propagating a trace context header from an HTTP response to an HTTP request:
|
||||
|
||||
**Traceparent header**
|
||||
|
||||
The traceparent header represents the incoming request in a tracing system in a common format, understood by all vendors:
|
||||
|
||||
```
|
||||
traceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01
|
||||
```
|
||||
|
||||
[Learn more about the traceparent fields details](https://www.w3.org/TR/trace-context/#traceparent-header).
|
||||
|
||||
**Tracestate header**
|
||||
|
||||
The tracestate header includes the parent in a potentially vendor-specific format:
|
||||
|
||||
```
|
||||
tracestate: congo=t61rcWkgMzE
|
||||
```
|
||||
|
||||
[Learn more about the tracestate fields details](https://www.w3.org/TR/trace-context/#tracestate-header).
|
||||
|
||||
{{% /codetab %}}
|
||||
|
||||
|
||||
<!-- gRPC -->
|
||||
{{% codetab %}}
|
||||
|
||||
In the gRPC API calls, trace context is passed through `grpc-trace-bin` header.
|
||||
|
||||
{{% /codetab %}}
|
||||
|
||||
{{< /tabs >}}
|
||||
|
||||
## Related Links
|
||||
- [Learn more about distributed tracing in Dapr]({{< ref tracing-overview.md >}})
|
||||
- [W3C Trace Context specification](https://www.w3.org/TR/trace-context/)
|
||||
|
|
@ -0,0 +1,110 @@
|
|||
---
|
||||
type: docs
|
||||
title: "How-To: Set up Zipkin for distributed tracing"
|
||||
linkTitle: "Zipkin"
|
||||
weight: 4000
|
||||
description: "Set up Zipkin for distributed tracing"
|
||||
type: docs
|
||||
---
|
||||
|
||||
## Configure self hosted mode
|
||||
|
||||
For self hosted mode, on running `dapr init`:
|
||||
|
||||
1. The following YAML file is created by default in `$HOME/.dapr/config.yaml` (on Linux/Mac) or `%USERPROFILE%\.dapr\config.yaml` (on Windows) and it is referenced by default on `dapr run` calls unless otherwise overridden `:
|
||||
|
||||
* config.yaml
|
||||
|
||||
```yaml
|
||||
apiVersion: dapr.io/v1alpha1
|
||||
kind: Configuration
|
||||
metadata:
|
||||
name: daprConfig
|
||||
namespace: default
|
||||
spec:
|
||||
tracing:
|
||||
samplingRate: "1"
|
||||
zipkin:
|
||||
endpointAddress: "http://localhost:9411/api/v2/spans"
|
||||
```
|
||||
|
||||
2. The [openzipkin/zipkin](https://hub.docker.com/r/openzipkin/zipkin/) docker container is launched on running `dapr init` or it can be launched with the following code.
|
||||
|
||||
Launch Zipkin using Docker:
|
||||
|
||||
```bash
|
||||
docker run -d -p 9411:9411 openzipkin/zipkin
|
||||
```
|
||||
|
||||
3. The applications launched with `dapr run` by default reference the config file in `$HOME/.dapr/config.yaml` or `%USERPROFILE%\.dapr\config.yaml` and can be overridden with the Dapr CLI using the `--config` param:
|
||||
|
||||
```bash
|
||||
dapr run --app-id mynode --app-port 3000 node app.js
|
||||
```
|
||||
### Viewing Traces
|
||||
To view traces, in your browser go to http://localhost:9411 and you will see the Zipkin UI.
|
||||
|
||||
## Configure Kubernetes
|
||||
|
||||
The following steps shows you how to configure Dapr to send distributed tracing data to Zipkin running as a container in your Kubernetes cluster, and how to view them.
|
||||
|
||||
### Setup
|
||||
|
||||
First, deploy Zipkin:
|
||||
|
||||
```bash
|
||||
kubectl create deployment zipkin --image openzipkin/zipkin
|
||||
```
|
||||
|
||||
Create a Kubernetes service for the Zipkin pod:
|
||||
|
||||
```bash
|
||||
kubectl expose deployment zipkin --type ClusterIP --port 9411
|
||||
```
|
||||
|
||||
Next, create the following YAML file locally:
|
||||
|
||||
* tracing.yaml configuration
|
||||
|
||||
```yaml
|
||||
apiVersion: dapr.io/v1alpha1
|
||||
kind: Configuration
|
||||
metadata:
|
||||
name: tracing
|
||||
namespace: default
|
||||
spec:
|
||||
tracing:
|
||||
samplingRate: "1"
|
||||
zipkin:
|
||||
endpointAddress: "http://zipkin.default.svc.cluster.local:9411/api/v2/spans"
|
||||
```
|
||||
|
||||
Now, deploy the the Dapr configuration file:
|
||||
|
||||
```bash
|
||||
kubectl apply -f tracing.yaml
|
||||
```
|
||||
|
||||
In order to enable this configuration for your Dapr sidecar, add the following annotation to your pod spec template:
|
||||
|
||||
```yml
|
||||
annotations:
|
||||
dapr.io/config: "tracing"
|
||||
```
|
||||
|
||||
That's it! Your sidecar is now configured to send traces to Zipkin.
|
||||
|
||||
### Viewing Tracing Data
|
||||
|
||||
To view traces, connect to the Zipkin service and open the UI:
|
||||
|
||||
```bash
|
||||
kubectl port-forward svc/zipkin 9411:9411
|
||||
```
|
||||
|
||||
In your browser, go to `http://localhost:9411` and you will see the Zipkin UI.
|
||||
|
||||
|
||||
|
||||
## References
|
||||
- [Zipkin for distributed tracing](https://zipkin.io/)
|
||||
Loading…
Reference in New Issue