mirror of https://github.com/dapr/docs.git
Update to observability docs for OTEL (#2876)
* otel doc Signed-off-by: msfussell <markfussell@gmail.com> * Update daprdocs/content/en/developing-applications/building-blocks/observability/tracing-overview.md Co-authored-by: Hannah Hunter <94493363+hhunter-ms@users.noreply.github.com> Signed-off-by: Mark Fussell <markfussell@gmail.com> * Update daprdocs/content/en/developing-applications/building-blocks/observability/tracing-overview.md Co-authored-by: Hannah Hunter <94493363+hhunter-ms@users.noreply.github.com> Signed-off-by: Mark Fussell <markfussell@gmail.com> * Update daprdocs/content/en/developing-applications/building-blocks/observability/tracing-overview.md Co-authored-by: Hannah Hunter <94493363+hhunter-ms@users.noreply.github.com> Signed-off-by: Mark Fussell <markfussell@gmail.com> * Update daprdocs/content/en/developing-applications/building-blocks/observability/tracing-overview.md Co-authored-by: Hannah Hunter <94493363+hhunter-ms@users.noreply.github.com> Signed-off-by: Mark Fussell <markfussell@gmail.com> * Update daprdocs/content/en/developing-applications/building-blocks/observability/tracing-overview.md Co-authored-by: Hannah Hunter <94493363+hhunter-ms@users.noreply.github.com> Signed-off-by: Mark Fussell <markfussell@gmail.com> * Update daprdocs/content/en/operations/monitoring/metrics/metrics-overview.md Co-authored-by: Hannah Hunter <94493363+hhunter-ms@users.noreply.github.com> Signed-off-by: Mark Fussell <markfussell@gmail.com> * Update daprdocs/content/en/operations/monitoring/tracing/otel-collector/_index.md Co-authored-by: Hannah Hunter <94493363+hhunter-ms@users.noreply.github.com> Signed-off-by: Mark Fussell <markfussell@gmail.com> * Update daprdocs/content/en/operations/monitoring/tracing/otel-collector/open-telemetry-collector.md Co-authored-by: Hannah Hunter <94493363+hhunter-ms@users.noreply.github.com> Signed-off-by: Mark Fussell <markfussell@gmail.com> * Update daprdocs/content/en/operations/monitoring/tracing/otel-collector/open-telemetry-collector.md Co-authored-by: Hannah Hunter <94493363+hhunter-ms@users.noreply.github.com> Signed-off-by: Mark Fussell <markfussell@gmail.com> * Update daprdocs/content/en/operations/monitoring/tracing/setup-tracing.md Co-authored-by: Yaron Schneider <schneider.yaron@live.com> Signed-off-by: Mark Fussell <markfussell@gmail.com> * Fixed URL address * Update daprdocs/content/en/developing-applications/building-blocks/observability/tracing-overview.md Co-authored-by: Hannah Hunter <94493363+hhunter-ms@users.noreply.github.com> Signed-off-by: Mark Fussell <markfussell@gmail.com> * Update daprdocs/content/en/developing-applications/building-blocks/observability/tracing-overview.md Co-authored-by: Hannah Hunter <94493363+hhunter-ms@users.noreply.github.com> Signed-off-by: Mark Fussell <markfussell@gmail.com> * Update daprdocs/content/en/developing-applications/building-blocks/observability/tracing-overview.md Co-authored-by: Hannah Hunter <94493363+hhunter-ms@users.noreply.github.com> Signed-off-by: Mark Fussell <markfussell@gmail.com> * Update daprdocs/content/en/developing-applications/building-blocks/observability/tracing-overview.md Co-authored-by: Hannah Hunter <94493363+hhunter-ms@users.noreply.github.com> Signed-off-by: Mark Fussell <markfussell@gmail.com> * Update daprdocs/content/en/developing-applications/building-blocks/observability/tracing-overview.md Co-authored-by: Hannah Hunter <94493363+hhunter-ms@users.noreply.github.com> Signed-off-by: Mark Fussell <markfussell@gmail.com> * Update daprdocs/content/en/developing-applications/building-blocks/observability/tracing-overview.md Co-authored-by: Hannah Hunter <94493363+hhunter-ms@users.noreply.github.com> Signed-off-by: Mark Fussell <markfussell@gmail.com> * Apply suggestions from code review Co-authored-by: Hannah Hunter <94493363+hhunter-ms@users.noreply.github.com> Signed-off-by: Mark Fussell <markfussell@gmail.com> * Update daprdocs/content/en/operations/monitoring/metrics/metrics-overview.md Co-authored-by: Hannah Hunter <94493363+hhunter-ms@users.noreply.github.com> Signed-off-by: Mark Fussell <markfussell@gmail.com> Signed-off-by: msfussell <markfussell@gmail.com> Signed-off-by: Mark Fussell <markfussell@gmail.com> Co-authored-by: Yaron Schneider <schneider.yaron@live.com> Co-authored-by: Hannah Hunter <94493363+hhunter-ms@users.noreply.github.com>
This commit is contained in:
Mark Fussell
GitHub
parent
8f08e681df
commit
4d860db7e1
|
|
@ -7,3 +7,10 @@ daprdocs/resources/_gen
|
|||
.venv/
|
||||
.hugo_build.lock
|
||||
.dccache
|
||||
.DS_Store
|
||||
daprdocs/.DS_Store
|
||||
daprdocs/content/.DS_Store
|
||||
daprdocs/content/en/.DS_Store
|
||||
daprdocs/resources/.DS_Store
|
||||
daprdocs/static/.DS_Store
|
||||
daprdocs/static/presentations/.DS_Store
|
||||
|
|
@ -4,41 +4,36 @@ title: "Observability"
|
|||
linkTitle: "Observability"
|
||||
weight: 500
|
||||
description: >
|
||||
Monitor applications through tracing, metrics, logs and health
|
||||
Observe applications through tracing, metrics, logs and health
|
||||
---
|
||||
|
||||
When building an application, understanding how the system is behaving is an important part of operating it - this includes having the ability to observe the internal calls of an application, gauging its performance and becoming aware of problems as soon as they occur. This is challenging for any system, but even more so for a distributed system comprised of multiple microservices where a flow, made of several calls, may start in one microservices but continue in another. Observability is critical in production environments, but also useful during development to understand bottlenecks, improve performance and perform basic debugging across the span of microservices.
|
||||
When building an application, understanding how the system is behaving is an important part of operating it - this includes having the ability to observe the internal calls of an application, gauging its performance and becoming aware of problems as soon as they occur. This is challenging for any system, but even more so for a distributed system comprised of multiple microservices where a flow, made of several calls, may start in one microservice but continue in another. Observability is critical in production environments, but also useful during development to understand bottlenecks, improve performance and perform basic debugging across the span of microservices.
|
||||
|
||||
While some data points about an application can be gathered from the underlying infrastructure (e.g. memory consumption, CPU usage), other meaningful information must be collected from an "application-aware" layer - one that can show how an important series of calls is executed across microservices. This usually means a developer must add some code to instrument an application for this purpose. Often, instrumentation code is simply meant to send collected data such as traces and metrics to an external monitoring tool or service that can help store, visualize and analyze all this information.
|
||||
While some data points about an application can be gathered from the underlying infrastructure (for example memory consumption, CPU usage), other meaningful information must be collected from an "application-aware" layer - one that can show how an important series of calls is executed across microservices. This usually means a developer must add some code to instrument an application for this purpose. Often, instrumentation code is simply meant to send collected data such as traces and metrics to observability tools or services that can help store, visualize and analyze all this information.
|
||||
|
||||
Having to maintain this code, which is not part of the core logic of the application, is another burden on the developer, sometimes requiring understanding the monitoring tools' APIs, using additional SDKs etc. This instrumentation may also add to the portability challenges of an application, which may require different instrumentation depending on where the application is deployed. For example, different cloud providers offer different monitoring solutions and an on-prem deployment might require an on-prem solution.
|
||||
Having to maintain this code, which is not part of the core logic of the application, is a burden on the developer, sometimes requiring understanding the observability tools' APIs, using additional SDKs etc. This instrumentation may also add to the portability challenges of an application, which may require different instrumentation depending on where the application is deployed. For example, different cloud providers offer different observability tools and an on-prem deployment might require an on-prem solution.
|
||||
|
||||
## Observability for your application with Dapr
|
||||
When building an application which leverages Dapr building blocks to perform service-to-service calls and pub/sub messaging, Dapr offers an advantage with respect to [distributed tracing]({{<ref tracing>}}). Because this inter-service communication flows through the Dapr sidecar, the sidecar is in a unique position to offload the burden of application-level instrumentation.
|
||||
When building an application which leverages Dapr API building blocks to perform service-to-service calls and pub/sub messaging, Dapr offers an advantage with respect to [distributed tracing]({{<ref tracing>}}). Because this inter-service communication flows through the Dapr sidecar, the sidecar is in a unique position to offload the burden of application-level instrumentation.
|
||||
|
||||
### Distributed tracing
|
||||
Dapr can be [configured to emit tracing data]({{<ref setup-tracing.md>}}), and because Dapr does so using widely adopted protocols such as the [Zipkin](https://zipkin.io) protocol, it can be easily integrated with multiple [monitoring backends]({{<ref supported-tracing-backends>}}).
|
||||
Dapr can be [configured to emit tracing data]({{<ref setup-tracing.md>}}), and because Dapr does so using the widely adopted protocols of [Open Telemetry (OTEL)](https://opentelemetry.io/) and [Zipkin](https://zipkin.io), it can be easily integrated with multiple observability tools.
|
||||
|
||||
<img src="/images/observability-tracing.png" width=1000 alt="Distributed tracing with Dapr">
|
||||
|
||||
### OpenTelemetry collector
|
||||
Dapr can also be configured to work with the [OpenTelemetry Collector]({{<ref open-telemetry-collector>}}) which offers even more compatibility with external monitoring tools.
|
||||
### Automatic tracing context generation
|
||||
Dapr uses [W3C tracing]({{<ref w3c-tracing-overview>}}) specification for tracing context, included as part Open Telemetry (OTEL), to generate and propagate the context header for the application or propagate user-provided context headers. This means that you get tracing by default with Dapr.
|
||||
|
||||
<img src="/images/observability-opentelemetry-collector.png" width=1000 alt="Distributed tracing via OpenTelemetry collector">
|
||||
|
||||
### Tracing context
|
||||
Dapr uses [W3C tracing]({{<ref w3c-tracing>}}) specification for tracing context and can generate and propagate the context header itself or propagate user-provided context headers.
|
||||
|
||||
## Observability for the Dapr sidecar and system services
|
||||
As for other parts of your system, you will want to be able to observe Dapr itself and collect metrics and logs emitted by the Dapr sidecar that runs along each microservice, as well as the Dapr-related services in your environment such as the control plane services that are deployed for a Dapr-enabled Kubernetes cluster.
|
||||
## Observability for the Dapr sidecar and control plane
|
||||
You also want to be able to observe Dapr itself, by collecting metrics on performance, throughput and latency and logs emitted by the Dapr sidecar, as well as the Dapr control plane services. Dapr sidecars have a health endpoint that can be probed to indicate their health status.
|
||||
|
||||
<img src="/images/observability-sidecar.png" width=1000 alt="Dapr sidecar metrics, logs and health checks">
|
||||
|
||||
### Logging
|
||||
Dapr generates [logs]({{<ref "logs.md">}}) to provide visibility into sidecar operation and to help users identify issues and perform debugging. Log events contain warning, error, info, and debug messages produced by Dapr system services. Dapr can also be configured to send logs to collectors such as [Fluentd]({{< ref fluentd.md >}}) and [Azure Monitor]({{< ref azure-monitor.md >}}) so they can be easily searched, analyzed and provide insights.
|
||||
Dapr generates [logs]({{<ref "logs.md">}}) to provide visibility into sidecar operation and to help users identify issues and perform debugging. Log events contain warning, error, info, and debug messages produced by Dapr system services. Dapr can also be configured to send logs to collectors such as [Fluentd]({{< ref fluentd.md >}}) and [Azure Monitor]({{< ref azure-monitor.md >}}) and others observability tools so they can be searched, analyzed and provide insights.
|
||||
|
||||
### Metrics
|
||||
Metrics are the series of measured values and counts that are collected and stored over time. [Dapr metrics]({{<ref "metrics">}}) provide monitoring capabilities to understand the behavior of the Dapr sidecar and system services. For example, the metrics between a Dapr sidecar and the user application show call latency, traffic failures, error rates of requests, etc. Dapr [system services metrics](https://github.com/dapr/dapr/blob/master/docs/development/dapr-metrics.md) show sidecar injection failures and the health of system services, including CPU usage, number of actor placements made, etc.
|
||||
Metrics are the series of measured values and counts that are collected and stored over time. [Dapr metrics]({{<ref "metrics">}}) provide monitoring capabilities to understand the behavior of the Dapr sidecar and control plane. For example, the metrics between a Dapr sidecar and the user application show call latency, traffic failures, error rates of requests, etc. Dapr [control plane metrics](https://github.com/dapr/dapr/blob/master/docs/development/dapr-metrics.md) show sidecar injection failures and the health of control plane services, including CPU usage, number of actor placements made, etc.
|
||||
|
||||
### Health checks
|
||||
The Dapr sidecar exposes an HTTP endpoint for [health checks]({{<ref sidecar-health.md>}}). With this API, user code or hosting environments can probe the Dapr sidecar to determine its status and identify issues with sidecar readiness.
|
||||
|
|
|
|||
|
|
@ -3,53 +3,113 @@ type: docs
|
|||
title: "Distributed tracing"
|
||||
linkTitle: "Distributed tracing"
|
||||
weight: 1000
|
||||
description: "Use Dapr tracing to get visibility for distributed application"
|
||||
description: "Use tracing to get visibility into your application"
|
||||
---
|
||||
|
||||
Dapr uses the Zipkin protocol for distributed traces and metrics collection. Due to the ubiquity of the Zipkin protocol, many backends are supported out of the box, for examples [Stackdriver](https://cloud.google.com/stackdriver), [Zipkin](https://zipkin.io), [New Relic](https://newrelic.com) and others. Combining with the OpenTelemetry Collector, Dapr can export traces to many other backends including but not limted to [Azure Monitor](https://azure.microsoft.com/services/monitor/), [Datadog](https://www.datadoghq.com), Instana, [Jaeger](https://www.jaegertracing.io/), and [SignalFX](https://www.signalfx.com/).
|
||||
Dapr uses the Open Telemetry (OTEL) and Zipkin protocols for distributed traces. OTEL is the industry standard and is the recommended trace protocol to use.
|
||||
|
||||
<img src="/images/tracing.png" width=600>
|
||||
Most observability tools support OTEL. For example [Google Cloud Operations](https://cloud.google.com/products/operations), [New Relic](https://newrelic.com), [Azure Monitor](https://azure.microsoft.com/services/monitor/), [Datadog](https://www.datadoghq.com), Instana, [Jaeger](https://www.jaegertracing.io/), and [SignalFX](https://www.signalfx.com/).
|
||||
|
||||
## Tracing design
|
||||
## Scenarios
|
||||
Tracing is used with service invocaton and pub/sub APIs. You can flow trace context between services that uses these APIs.
|
||||
|
||||
Dapr adds a HTTP/gRPC middleware to the Dapr sidecar. The middleware intercepts all Dapr and application traffic and automatically injects correlation IDs to trace distributed transactions. This design has several benefits:
|
||||
There are two scenarios for how tracing is used:
|
||||
1. Dapr generates the trace context and you propagate the trace context to another service.
|
||||
2. You generate the trace context and Dapr propagates the trace context to a service.
|
||||
|
||||
* No need for code instrumentation. All traffic is automatically traced with configurable tracing levels.
|
||||
* Consistent tracing behavior across microservices. Tracing is configured and managed on Dapr sidecar so that it remains consistent across services made by different teams and potentially written in different programming languages.
|
||||
* Configurable and extensible. By leveraging the Zipkin API and the OpenTelemetry Collector, Dapr tracing can be configured to work with popular tracing backends, including custom backends a customer may have.
|
||||
* You can define and enable multiple exporters at the same time.
|
||||
### Propogating sequential service calls
|
||||
Dapr takes care of creating the trace headers. However, when there are more than two services, you're responsible for propagating the trace headers between them. Let's go through the scenarios with examples:
|
||||
|
||||
## W3C Correlation ID
|
||||
1. Single service invocation call (`service A -> service B`)
|
||||
|
||||
Dapr uses the standard W3C Trace Context headers. For HTTP requests, Dapr uses `traceparent` header. For gRPC requests, Dapr uses `grpc-trace-bin` header. When a request arrives without a trace ID, Dapr creates a new one. Otherwise, it passes the trace ID along the call chain.
|
||||
Dapr generates the trace headers in service A, which are then propagated from service A to service B. No further propagation is needed.
|
||||
|
||||
Read [W3C distributed tracing]({{< ref w3c-tracing >}}) for more background on W3C Trace Context.
|
||||
2. Multiple sequential service invocation calls ( `service A -> service B -> service C`)
|
||||
|
||||
## Configuration
|
||||
Dapr generates the trace headers at the beginning of the request in service A, which are then propagated to service B. You are now responsible for taking the headers and propagating them to service C, since this is specific to your application.
|
||||
|
||||
Dapr uses probabilistic sampling. The sample rate defines the probability a tracing span will be sampled and can have a value between 0 and 1 (inclusive). The default sample rate is 0.0001 (i.e. 1 in 10,000 spans is sampled).
|
||||
`service A -> service B -> propagate trace headers to -> service C` and so on to further Dapr-enabled services.
|
||||
|
||||
To change the default tracing behavior, use a configuration file (in self hosted mode) or a Kubernetes configuration object (in Kubernetes mode). For example, the following configuration object changes the sample rate to 1 (i.e. every span is sampled), and sends trace using Zipkin protocol to the Zipkin server at http://zipkin.default.svc.cluster.local
|
||||
In other words, if the app is calling to Dapr and wants to trace with an existing span (trace header), it must always propagate to Dapr (from service B to service C in this case). Dapr always propagates trace spans to an application.
|
||||
|
||||
```yaml
|
||||
apiVersion: dapr.io/v1alpha1
|
||||
kind: Configuration
|
||||
metadata:
|
||||
name: tracing
|
||||
namespace: default
|
||||
spec:
|
||||
tracing:
|
||||
samplingRate: "1"
|
||||
zipkin:
|
||||
endpointAddress: "http://zipkin.default.svc.cluster.local:9411/api/v2/spans"
|
||||
```
|
||||
{{% alert title="Note" color="primary" %}}
|
||||
There are no helper methods exposed in Dapr SDKs to propagate and retrieve trace context. You need to use HTTP/gRPC clients to propagate and retrieve trace headers through HTTP headers and gRPC metadata.
|
||||
{{% /alert %}}
|
||||
|
||||
Note: Changing `samplingRate` to 0 disables tracing altogether.
|
||||
3. Request is from external endpoint (for example, `from a gateway service to a Dapr-enabled service A`)
|
||||
|
||||
See the [References](#references) section for more details on how to configure tracing on local environment and Kubernetes environment.
|
||||
An external gateway ingress calls Dapr, which generates the trace headers and calls service A. Service A then calls service B and further Dapr-enabled services. You must propagate the headers from service A to service B: `Ingress -> service A -> propagate trace headers -> service B`. This is similar to case 2 above.
|
||||
|
||||
## References
|
||||
4. Pub/sub messages
|
||||
Dapr generates the trace headers in the published message topic. These trace headers are propagated to any services listening on that topic.
|
||||
|
||||
- [How-To: Setup Application Insights for distributed tracing with OpenTelemetry Collector]({{< ref open-telemetry-collector.md >}})
|
||||
- [How-To: Set up Zipkin for distributed tracing]({{< ref zipkin.md >}})
|
||||
- [W3C distributed tracing]({{< ref w3c-tracing >}})
|
||||
### Propogating multiple different service calls
|
||||
In the following scenarios, Dapr does some of the work for you and you need to either create or propagate trace headers.
|
||||
|
||||
1. Multiple service calls to different services from single service
|
||||
|
||||
When you are calling multiple services from a single service (see example below), you need to propagate the trace headers:
|
||||
|
||||
```
|
||||
service A -> service B
|
||||
[ .. some code logic ..]
|
||||
service A -> service C
|
||||
[ .. some code logic ..]
|
||||
service A -> service D
|
||||
[ .. some code logic ..]
|
||||
```
|
||||
|
||||
In this case, when service A first calls service B, Dapr generates the trace headers in service A, which are then propagated to service B. These trace headers are returned in the response from service B as part of response headers. You then need to propagate the returned trace context to the next services, service C and service D, as Dapr does not know you want to reuse the same header.
|
||||
|
||||
### Generating your own trace context headers from non-Daprized applications
|
||||
|
||||
You may have chosen to generate your own trace context headers.
|
||||
Generating your own trace context headers is more unusual and typically not required when calling Dapr. However, there are scenarios where you could specifically choose to add W3C trace headers into a service call; for example, you have an existing application that does not use Dapr. In this case, Dapr still propagates the trace context headers for you. If you decide to generate trace headers yourself, there are three ways this can be done:
|
||||
|
||||
1. You can use the industry standard [OpenTelemetry SDKs](https://opentelemetry.io/docs/instrumentation/) to generate trace headers and pass these trace headers to a Dapr-enabled service. This is the preferred method.
|
||||
|
||||
2. You can use a vendor SDK that provides a way to generate W3C trace headers and pass them to a Dapr-enabled service.
|
||||
|
||||
3. You can handcraft a trace context following [W3C trace context specifications](https://www.w3.org/TR/trace-context/) and pass them to a Dapr-enabled service.
|
||||
|
||||
## W3C trace context
|
||||
|
||||
Dapr uses the standard W3C trace context headers.
|
||||
|
||||
- For HTTP requests, Dapr uses `traceparent` header.
|
||||
- For gRPC requests, Dapr uses `grpc-trace-bin` header.
|
||||
|
||||
When a request arrives without a trace ID, Dapr creates a new one. Otherwise, it passes the trace ID along the call chain.
|
||||
|
||||
Read [trace context overview]({{< ref w3c-tracing-overview >}}) for more background on W3C trace context.
|
||||
|
||||
## W3C trace headers
|
||||
These are the specific trace context headers that are generated and propagated by Dapr for HTTP and gRPC.
|
||||
|
||||
### Trace context HTTP headers format
|
||||
When propagating a trace context header from an HTTP response to an HTTP request, you copy these headers.
|
||||
|
||||
#### Traceparent header
|
||||
The traceparent header represents the incoming request in a tracing system in a common format, understood by all vendors.
|
||||
Here’s an example of a traceparent header.
|
||||
|
||||
`traceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01`
|
||||
|
||||
Find the traceparent fields detailed [here](https://www.w3.org/TR/trace-context/#traceparent-header).
|
||||
|
||||
#### Tracestate header
|
||||
The tracestate header includes the parent in a potentially vendor-specific format:
|
||||
|
||||
`tracestate: congo=t61rcWkgMzE`
|
||||
|
||||
Find the tracestate fields detailed [here](https://www.w3.org/TR/trace-context/#tracestate-header).
|
||||
|
||||
### Trace context gRPC headers format
|
||||
In the gRPC API calls, trace context is passed through `grpc-trace-bin` header.
|
||||
|
||||
## Related Links
|
||||
|
||||
- [Observability concepts]({{< ref observability-concept.md >}})
|
||||
- [W3C Trace Context for distributed tracing]({{< ref w3c-tracing-overview >}})
|
||||
- [W3C Trace Context specification](https://www.w3.org/TR/trace-context/)
|
||||
- [Observability quickstart](https://github.com/dapr/quickstarts/tree/master/tutorials/observability)
|
||||
|
|
@ -0,0 +1,33 @@
|
|||
---
|
||||
type: docs
|
||||
title: "Trace context"
|
||||
linkTitle: "Trace context"
|
||||
weight: 4000
|
||||
description: Background and scenarios for using W3C tracing with Dapr
|
||||
type: docs
|
||||
---
|
||||
|
||||
Dapr uses the [Open Telemetry protocol](https://opentelemetry.io/), which in turn uses the [W3C trace context](https://www.w3.org/TR/trace-context/) for distributed tracing for both service invocation and pub/sub messaging. Dapr generates and propagates the trace context information, which can be sent to observability tools for visualization and querying.
|
||||
|
||||
## Background
|
||||
Distributed tracing is a methodology implemented by tracing tools to follow, analyze, and debug a transaction across multiple software components. Typically, a distributed trace traverses more than one service which requires it to be uniquely identifiable. Trace context propagation passes along this unique identification.
|
||||
|
||||
In the past, trace context propagation has typically been implemented individually by each different tracing vendor. In multi-vendor environments, this causes interoperability problems, such as:
|
||||
|
||||
- Traces that are collected by different tracing vendors cannot be correlated as there is no shared unique identifier.
|
||||
- Traces that cross boundaries between different tracing vendors can not be propagated as there is no forwarded, uniformly agreed set of identification.
|
||||
- Vendor-specific metadata might be dropped by intermediaries.
|
||||
- Cloud platform vendors, intermediaries, and service providers cannot guarantee to support trace context propagation as there is no standard to follow.
|
||||
|
||||
In the past, these problems did not have a significant impact, as most applications were monitored by a single tracing vendor and stayed within the boundaries of a single platform provider. Today, an increasing number of applications are distributed and leverage multiple middleware services and cloud platforms.
|
||||
|
||||
This transformation of modern applications called for a distributed tracing context propagation standard. The [W3C trace context specification](https://www.w3.org/TR/trace-context/) defines a universally agreed-upon format for the exchange of trace context propagation data - referred to as trace context. Trace context solves the problems described above by:
|
||||
|
||||
* Providing a unique identifier for individual traces and requests, allowing trace data of multiple providers to be linked together.
|
||||
* Providing an agreed-upon mechanism to forward vendor-specific trace data and avoid broken traces when multiple tracing tools participate in a single transaction.
|
||||
* Providing an industry standard that intermediaries, platforms, and hardware providers can support.
|
||||
|
||||
A unified approach for propagating trace data improves visibility into the behavior of distributed applications, facilitating problem and performance analysis.
|
||||
|
||||
## Related Links
|
||||
- [W3C Trace Context specification](https://www.w3.org/TR/trace-context/)
|
||||
|
|
@ -1,8 +0,0 @@
|
|||
---
|
||||
type: docs
|
||||
title: "W3C trace context"
|
||||
linkTitle: "W3C trace context"
|
||||
weight: 1000
|
||||
description: Background and scenarios for using W3C tracing with Dapr
|
||||
type: docs
|
||||
---
|
||||
|
|
@ -1,297 +0,0 @@
|
|||
---
|
||||
type: docs
|
||||
title: "How-To: Use W3C trace context with Dapr"
|
||||
linkTitle: "How-To: Use W3C trace context"
|
||||
weight: 20000
|
||||
description: Using W3C tracing standard with Dapr
|
||||
type: docs
|
||||
---
|
||||
|
||||
# How to use trace context
|
||||
Dapr uses W3C trace context for distributed tracing for both service invocation and pub/sub messaging. Dapr does all the heavy lifting of generating and propagating the trace context information and there are very few cases where you need to either propagate or create a trace context. First read scenarios in the [W3C distributed tracing]({{< ref w3c-tracing >}}) article to understand whether you need to propagate or create a trace context.
|
||||
|
||||
To view traces, read the [how to diagnose with tracing]({{< ref tracing-overview.md >}}) article.
|
||||
|
||||
## How to retrieve trace context from a response
|
||||
`Note: There are no helper methods exposed in Dapr SDKs to propagate and retrieve trace context. You need to use http/gRPC clients to propagate and retrieve trace headers through http headers and gRPC metadata.`
|
||||
|
||||
### Retrieve trace context in Go
|
||||
#### For HTTP calls
|
||||
OpenCensus Go SDK provides [ochttp](https://pkg.go.dev/go.opencensus.io/plugin/ochttp/propagation/tracecontext?tab=doc) package that provides methods to retrieve trace context from http response.
|
||||
|
||||
To retrieve the trace context from HTTP response, you can use :
|
||||
|
||||
```go
|
||||
f := tracecontext.HTTPFormat{}
|
||||
sc, ok := f.SpanContextFromRequest(req)
|
||||
```
|
||||
#### For gRPC calls
|
||||
To retrieve the trace context header when the gRPC call is returned, you can pass the response header reference as gRPC call option which contains response headers:
|
||||
|
||||
```go
|
||||
var responseHeader metadata.MD
|
||||
|
||||
// Call the InvokeService with call option
|
||||
// grpc.Header(&responseHeader)
|
||||
|
||||
client.InvokeService(ctx, &pb.InvokeServiceRequest{
|
||||
Id: "client",
|
||||
Message: &commonv1pb.InvokeRequest{
|
||||
Method: "MyMethod",
|
||||
ContentType: "text/plain; charset=UTF-8",
|
||||
Data: &any.Any{Value: []byte("Hello")},
|
||||
},
|
||||
},
|
||||
grpc.Header(&responseHeader))
|
||||
```
|
||||
|
||||
### Retrieve trace context in C#
|
||||
#### For HTTP calls
|
||||
To retrieve the trace context from HTTP response, you can use [.NET API](https://docs.microsoft.com/dotnet/api/system.net.http.headers.httpresponseheaders?view=netcore-3.1) :
|
||||
|
||||
```csharp
|
||||
// client is HttpClient. req is HttpRequestMessage
|
||||
HttpResponseMessage response = await client.SendAsync(req);
|
||||
IEnumerable<string> values1, values2;
|
||||
string traceparentValue = "";
|
||||
string tracestateValue = "";
|
||||
if (response.Headers.TryGetValues("traceparent", out values1))
|
||||
{
|
||||
traceparentValue = values1.FirstOrDefault();
|
||||
}
|
||||
if (response.Headers.TryGetValues("tracestate", out values2))
|
||||
{
|
||||
tracestateValue = values2.FirstOrDefault();
|
||||
}
|
||||
```
|
||||
|
||||
#### For gRPC calls
|
||||
To retrieve the trace context from gRPC response, you can use [Grpc.Net.Client](https://www.nuget.org/packages/Grpc.Net.Client) ResponseHeadersAsync method.
|
||||
|
||||
```csharp
|
||||
// client is Dapr proto client
|
||||
using var call = client.InvokeServiceAsync(req);
|
||||
var response = await call.ResponseAsync;
|
||||
var headers = await call.ResponseHeadersAsync();
|
||||
var tracecontext = headers.First(e => e.Key == "grpc-trace-bin");
|
||||
```
|
||||
Additional general details on calling gRPC services with .NET client [here](https://docs.microsoft.com/aspnet/core/grpc/client?view=aspnetcore-3.1).
|
||||
|
||||
## How to propagate trace context in a request
|
||||
`Note: There are no helper methods exposed in Dapr SDKs to propagate and retrieve trace context. You need to use http/gRPC clients to propagate and retrieve trace headers through http headers and gRPC metadata.`
|
||||
|
||||
### Pass trace context in Go
|
||||
#### For HTTP calls
|
||||
OpenCensus Go SDK provides [ochttp](https://pkg.go.dev/go.opencensus.io/plugin/ochttp/propagation/tracecontext?tab=doc) package that provides methods to attach trace context in http request.
|
||||
|
||||
```go
|
||||
f := tracecontext.HTTPFormat{}
|
||||
req, _ := http.NewRequest("GET", "http://localhost:3500/v1.0/invoke/mathService/method/api/v1/add", nil)
|
||||
|
||||
traceContext := span.SpanContext()
|
||||
f.SpanContextToRequest(traceContext, req)
|
||||
```
|
||||
|
||||
#### For gRPC calls
|
||||
|
||||
```go
|
||||
traceContext := span.SpanContext()
|
||||
traceContextBinary := propagation.Binary(traceContext)
|
||||
```
|
||||
|
||||
You can then pass the trace context through [gRPC metadata](https://google.golang.org/grpc/metadata) through `grpc-trace-bin` header.
|
||||
|
||||
```go
|
||||
ctx = metadata.AppendToOutgoingContext(ctx, "grpc-trace-bin", string(traceContextBinary))
|
||||
```
|
||||
|
||||
You can then continuing passing this go context `ctx` in subsequent Dapr gRPC calls as first parameter. For example `InvokeService`, context is passed in first parameter.
|
||||
|
||||
### Pass trace context in C#
|
||||
#### For HTTP calls
|
||||
To pass trace context in HTTP request, you can use [.NET API](https://docs.microsoft.com/dotnet/api/system.net.http.headers.httprequestheaders?view=netcore-3.1) :
|
||||
|
||||
```csharp
|
||||
// client is HttpClient. req is HttpRequestMessage
|
||||
req.Headers.Add("traceparent", traceparentValue);
|
||||
req.Headers.Add("tracestate", tracestateValue);
|
||||
HttpResponseMessage response = await client.SendAsync(req);
|
||||
```
|
||||
|
||||
#### For gRPC calls
|
||||
To pass the trace context in gRPC call metadata, you can use [Grpc.Net.Client](https://www.nuget.org/packages/Grpc.Net.Client) ResponseHeadersAsync method.
|
||||
|
||||
```csharp
|
||||
// client is Dapr.Client.Autogen.Grpc.v1
|
||||
var headers = new Metadata();
|
||||
headers.Add("grpc-trace-bin", tracecontext);
|
||||
using var call = client.InvokeServiceAsync(req, headers);
|
||||
```
|
||||
Additional general details on calling gRPC services with .NET client [here](https://docs.microsoft.com/aspnet/core/grpc/client?view=aspnetcore-3.1).
|
||||
|
||||
## How to create trace context
|
||||
You can create a trace context using the recommended OpenCensus SDKs. OpenCensus supports several different programming languages.
|
||||
|
||||
| Language | SDK |
|
||||
|:-------:|:----:|
|
||||
| Go | [Link](https://pkg.go.dev/go.opencensus.io?tab=overview)
|
||||
| Java | [Link](https://github.com/census-instrumentation/opencensus-java)
|
||||
| C# | [Link](https://github.com/census-instrumentation/opencensus-csharp/)
|
||||
| C++ | [Link](https://github.com/census-instrumentation/opencensus-cpp)
|
||||
| Node.js | [Link](https://github.com/census-instrumentation/opencensus-node)
|
||||
| Python | [Link](https://census-instrumentation.github.io/opencensus-python/trace/api/index.html)
|
||||
|
||||
### Create trace context in Go
|
||||
|
||||
#### 1. Get the OpenCensus Go SDK
|
||||
|
||||
Prerequisites: OpenCensus Go libraries require Go 1.8 or later. For details on installation go [here](https://pkg.go.dev/go.opencensus.io?tab=overview).
|
||||
|
||||
#### 2. Import the package "go.opencensus.io/trace"
|
||||
`$ go get -u go.opencensus.io`
|
||||
|
||||
#### 3. Create trace context
|
||||
|
||||
```go
|
||||
ctx, span := trace.StartSpan(ctx, "cache.Get")
|
||||
defer span.End()
|
||||
|
||||
// Do work to get from cache.
|
||||
```
|
||||
|
||||
### Create trace context in Java
|
||||
|
||||
```java
|
||||
try (Scope ss = TRACER.spanBuilder("cache.Get").startScopedSpan()) {
|
||||
}
|
||||
```
|
||||
|
||||
### Create trace context in Python
|
||||
|
||||
```python
|
||||
with tracer.span(name="cache.get") as span:
|
||||
pass
|
||||
```
|
||||
|
||||
### Create trace context in NodeJS
|
||||
|
||||
```nodejs
|
||||
tracer.startRootSpan({name: 'cache.Get'}, rootSpan => {
|
||||
});
|
||||
```
|
||||
|
||||
### Create trace context in C++
|
||||
|
||||
```cplusplus
|
||||
opencensus::trace::Span span = opencensus::trace::Span::StartSpan(
|
||||
"cache.Get", nullptr, {&sampler});
|
||||
```
|
||||
|
||||
### Create trace context in C#
|
||||
|
||||
```csharp
|
||||
var span = tracer.SpanBuilder("cache.Get").StartScopedSpan();
|
||||
```
|
||||
|
||||
## Putting it all together with a Go Sample
|
||||
|
||||
### Configure tracing in Dapr
|
||||
First you need to enable tracing configuration in Dapr. This step is mentioned for completeness from enabling tracing to invoking Dapr with trace context.
|
||||
Create a deployment config yaml e.g. `appconfig.yaml` with following configuration.
|
||||
|
||||
```yaml
|
||||
apiVersion: dapr.io/v1alpha1
|
||||
kind: Configuration
|
||||
metadata:
|
||||
name: appconfig
|
||||
spec:
|
||||
tracing:
|
||||
samplingRate: "1"
|
||||
```
|
||||
|
||||
In Kubernetes, you can apply the configuration as below :
|
||||
|
||||
```bash
|
||||
kubectl apply -f appconfig.yaml
|
||||
```
|
||||
|
||||
You then set the following tracing annotation in your deployment YAML. You can add the following annotaion in sample [grpc app]({{< ref grpc.md >}}) deployment yaml.
|
||||
|
||||
```yaml
|
||||
dapr.io/config: "appconfig"
|
||||
```
|
||||
|
||||
### Invoking Dapr with trace context
|
||||
|
||||
Dapr covers generating trace context and you do not need to explicitly create trace context.
|
||||
|
||||
However if you choose to pass the trace context explicitly, then Dapr will use the passed trace context and propagate all across the HTTP/gRPC call.
|
||||
|
||||
Using the [grpc app]({{< ref grpc.md >}}) in the example and putting this all together, the following steps show you how to create a Dapr client and call the InvokeService method passing the trace context:
|
||||
|
||||
The Rest code snippet and details, refer to the [grpc app]({{< ref grpc >}}).
|
||||
|
||||
### 1. Import the package
|
||||
|
||||
```go
|
||||
package main
|
||||
|
||||
import (
|
||||
pb "github.com/dapr/go-sdk/dapr"
|
||||
"go.opencensus.io/trace"
|
||||
"go.opencensus.io/trace/propagation"
|
||||
"google.golang.org/grpc"
|
||||
"google.golang.org/grpc/metadata"
|
||||
)
|
||||
```
|
||||
|
||||
### 2. Create the client
|
||||
|
||||
```go
|
||||
// Get the Dapr port and create a connection
|
||||
daprPort := os.Getenv("DAPR_GRPC_PORT")
|
||||
daprAddress := fmt.Sprintf("localhost:%s", daprPort)
|
||||
conn, err := grpc.Dial(daprAddress, grpc.WithInsecure())
|
||||
if err != nil {
|
||||
fmt.Println(err)
|
||||
}
|
||||
defer conn.Close()
|
||||
|
||||
// Create the client
|
||||
client := pb.NewDaprClient(conn)
|
||||
```
|
||||
|
||||
### 3. Invoke the InvokeService method With Trace Context
|
||||
|
||||
```go
|
||||
// Create the Trace Context
|
||||
ctx , span := trace.StartSpan(context.Background(), "InvokeService")
|
||||
|
||||
// The returned context can be used to keep propagating the newly created span in the current context.
|
||||
// In the same process, context.Context is used to propagate trace context.
|
||||
|
||||
// Across the process, use the propagation format of Trace Context to propagate trace context.
|
||||
traceContext := propagation.Binary(span.SpanContext())
|
||||
ctx = metadata.NewOutgoingContext(ctx, string(traceContext))
|
||||
|
||||
// Pass the trace context
|
||||
resp, err := client.InvokeService(ctx, &pb.InvokeServiceRequest{
|
||||
Id: "client",
|
||||
Message: &commonv1pb.InvokeRequest{
|
||||
Method: "MyMethod",
|
||||
ContentType: "text/plain; charset=UTF-8",
|
||||
Data: &any.Any{Value: []byte("Hello")},
|
||||
},
|
||||
})
|
||||
```
|
||||
|
||||
You can now correlate the calls in your app and across services with Dapr using the same trace context.
|
||||
|
||||
## Related Links
|
||||
|
||||
- [Observability concepts]({{< ref observability-concept.md >}})
|
||||
- [W3C Trace Context for distributed tracing]({{< ref w3c-tracing >}})
|
||||
- [How To set up Application Insights for distributed tracing with OpenTelemetry]({{< ref open-telemetry-collector.md >}})
|
||||
- [How to set up Zipkin for distributed tracing]({{< ref zipkin.md >}})
|
||||
- [W3C trace context specification](https://www.w3.org/TR/trace-context/)
|
||||
- [Observability quickstart](https://github.com/dapr/quickstarts/tree/master/tutorials/observability)
|
||||
|
|
@ -1,111 +0,0 @@
|
|||
---
|
||||
type: docs
|
||||
title: "W3C trace context overview"
|
||||
linkTitle: "Overview"
|
||||
weight: 10000
|
||||
description: Background and scenarios for using W3C tracing with Dapr
|
||||
type: docs
|
||||
---
|
||||
|
||||
## Introduction
|
||||
Dapr uses W3C trace context for distributed tracing for both service invocation and pub/sub messaging. Largely Dapr does all the heavy lifting of generating and propogating the trace context information and this can be sent to many different diagnostics tools for visualization and querying. There are only a very few cases where you, as a developer, need to either propagate a trace header or generate one.
|
||||
|
||||
## Background
|
||||
Distributed tracing is a methodology implemented by tracing tools to follow, analyze and debug a transaction across multiple software components. Typically, a distributed trace traverses more than one service which requires it to be uniquely identifiable. Trace context propagation passes along this unique identification.
|
||||
|
||||
In the past, trace context propagation has typically been implemented individually by each different tracing vendors. In multi-vendor environments, this causes interoperability problems, such as;
|
||||
|
||||
- Traces that are collected by different tracing vendors cannot be correlated as there is no shared unique identifier.
|
||||
- Traces that cross boundaries between different tracing vendors can not be propagated as there is no uniformly agreed set of identification that is forwarded.
|
||||
- Vendor specific metadata might be dropped by intermediaries.
|
||||
- Cloud platform vendors, intermediaries and service providers, cannot guarantee to support trace context propagation as there is no standard to follow.
|
||||
|
||||
In the past, these problems did not have a significant impact as most applications were monitored by a single tracing vendor and stayed within the boundaries of a single platform provider. Today, an increasing number of applications are distributed and leverage multiple middleware services and cloud platforms.
|
||||
|
||||
This transformation of modern applications called for a distributed tracing context propagation standard. The [W3C trace context specification](https://www.w3.org/TR/trace-context/) defines a universally agreed-upon format for the exchange of trace context propagation data - referred to as trace context. Trace context solves the problems described above by;
|
||||
|
||||
* Providing an unique identifier for individual traces and requests, allowing trace data of multiple providers to be linked together.
|
||||
* Providing an agreed-upon mechanism to forward vendor-specific trace data and avoid broken traces when multiple tracing tools participate in a single transaction.
|
||||
* Providing an industry standard that intermediaries, platforms, and hardware providers can support.
|
||||
|
||||
A unified approach for propagating trace data improves visibility into the behavior of distributed applications, facilitating problem and performance analysis.
|
||||
|
||||
## Scenarios
|
||||
There are two scenarios where you need to understand how tracing is used:
|
||||
1. Dapr generates and propagates the trace context between services.
|
||||
2. Dapr generates the trace context and you need to propagate the trace context to another service **or** you generate the trace context and Dapr propagates the trace context to a service.
|
||||
|
||||
### Dapr generates and propagates the trace context between services.
|
||||
In these scenarios Dapr does all work for you. You do not need to create and propagate any trace headers. Dapr takes care of creating all trace headers and propogating them. Let's go through the scenarios with examples;
|
||||
|
||||
1. Single service invocation call (`service A -> service B` )
|
||||
|
||||
Dapr generates the trace headers in service A and these trace headers are propagated from service A to service B.
|
||||
|
||||
2. Multiple sequential service invocation calls ( `service A -> service B -> service C`)
|
||||
|
||||
Dapr generates the trace headers at the beginning of the request in service A and these trace headers are propagated from `service A-> service B -> service C` and so on to further Dapr enabled services.
|
||||
|
||||
3. Request is from external endpoint (`For example from a gateway service to a Dapr enabled service A`)
|
||||
|
||||
Dapr generates the trace headers in service A and these trace headers are propagated from service A to further Dapr enabled services `service A-> service B -> service C`. This is similar to above case 2.
|
||||
|
||||
4. Pub/sub messages
|
||||
Dapr generates the trace headers in the published message topic and these trace headers are propagated to any services listening on that topic.
|
||||
|
||||
### You need to propagate or generate trace context between services
|
||||
In these scenarios Dapr does some of the work for you and you need to either create or propagate trace headers.
|
||||
|
||||
1. Multiple service calls to different services from single service
|
||||
|
||||
When you are calling multiple services from a single service, for example from service A like this, you need to propagate the trace headers;
|
||||
|
||||
service A -> service B
|
||||
[ .. some code logic ..]
|
||||
service A -> service C
|
||||
[ .. some code logic ..]
|
||||
service A -> service D
|
||||
[ .. some code logic ..]
|
||||
|
||||
In this case, when service A first calls service B, Dapr generates the trace headers in service A, and these trace headers are then propagated to service B. These trace headers are returned in the response from service B as part of response headers. However you need to propagate the returned trace context to the next services, service C and Service D, as Dapr does not know you want to reuse the same header.
|
||||
|
||||
To understand how to extract the trace headers from a response and add the trace headers into a request, see the [how to use trace context]({{< ref w3c-tracing-howto >}}) article.
|
||||
|
||||
2. You have chosen to generate your own trace context headers.
|
||||
This is much more unusual. There may be occasions where you specifically chose to add W3C trace headers into a service call, for example if you have an existing application that does not currently use Dapr. In this case Dapr still propagates the trace context headers for you. If you decide to generate trace headers yourself, there are three ways this can be done :
|
||||
|
||||
1. You can use the industry standard OpenCensus/OpenTelemetry SDKs to generate trace headers and pass these trace headers to a Dapr enabled service. This is the preferred recommendation.
|
||||
|
||||
2. You can use a vendor SDK that provides a way to generate W3C trace headers such as DynaTrace SDK and pass these trace headers to a Dapr enabled service.
|
||||
|
||||
3. You can handcraft a trace context following [W3C trace context specification](https://www.w3.org/TR/trace-context/) and pass these trace headers to Dapr enabled service.
|
||||
|
||||
## W3C trace headers
|
||||
Theses are the specific trace context headers that are generated and propagated by Dapr for HTTP and gRPC.
|
||||
|
||||
### Trace context HTTP headers format
|
||||
When propagating a trace context header from an HTTP response to an HTTP request, these are the headers that you need to copy.
|
||||
|
||||
#### Traceparent Header
|
||||
The traceparent header represents the incoming request in a tracing system in a common format, understood by all vendors.
|
||||
Here’s an example of a traceparent header.
|
||||
|
||||
`traceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01`
|
||||
|
||||
The traceparent fields are detailed [here](https://www.w3.org/TR/trace-context/#traceparent-header)
|
||||
|
||||
#### Tracestate Header
|
||||
The tracestate header includes the parent in a potentially vendor-specific format:
|
||||
|
||||
`tracestate: congo=t61rcWkgMzE`
|
||||
|
||||
The tracestate fields are detailed [here](https://www.w3.org/TR/trace-context/#tracestate-header)
|
||||
|
||||
### Trace context gRPC headers format
|
||||
In the gRPC API calls, trace context is passed through `grpc-trace-bin` header.
|
||||
|
||||
## Related Links
|
||||
- [How To set up Application Insights for distributed tracing with OpenTelemetry]({{< ref open-telemetry-collector.md >}})
|
||||
- [How To set up Zipkin for distributed tracing]({{< ref zipkin.md >}})
|
||||
- [W3C trace context specification](https://www.w3.org/TR/trace-context/)
|
||||
- [Observability sample](https://github.com/dapr/quickstarts/tree/master/tutorials/observability)
|
||||
|
|
@ -65,6 +65,8 @@ The `tracing` section under the `Configuration` spec contains the following prop
|
|||
```yml
|
||||
tracing:
|
||||
samplingRate: "1"
|
||||
otel:
|
||||
endpointAddress: "https://..."
|
||||
zipkin:
|
||||
endpointAddress: "http://zipkin.default.svc.cluster.local:9411/api/v2/spans"
|
||||
```
|
||||
|
|
@ -74,7 +76,11 @@ The following table lists the properties for tracing:
|
|||
| Property | Type | Description |
|
||||
|--------------|--------|-------------|
|
||||
| `samplingRate` | string | Set sampling rate for tracing to be enabled or disabled.
|
||||
| `zipkin.endpointAddress` | string | Set the Zipkin server address.
|
||||
| `stdout` | bool | True write more verbose information to the traces
|
||||
| `otel.endpointAddress` | string | Set the Open Telemetry (OTEL) server address to send traces to
|
||||
| `otel.isSecure` | bool | Is the connection to the endpoint address encryped
|
||||
| `otel.protocol` | string | Set to `http` or `grpc` protocol
|
||||
| `zipkin.endpointAddress` | string | Set the Zipkin server address to send traces to
|
||||
|
||||
`samplingRate` is used to enable or disable the tracing. To disable the sampling rate ,
|
||||
set `samplingRate : "0"` in the configuration. The valid range of samplingRate is between 0 and 1 inclusive. The sampling rate determines whether a trace span should be sampled or not based on value. `samplingRate : "1"` samples all traces. By default, the sampling rate is (0.0001) or 1 in 10,000 traces.
|
||||
|
|
@ -180,6 +186,11 @@ metadata:
|
|||
spec:
|
||||
tracing:
|
||||
samplingRate: "1"
|
||||
stdout: true
|
||||
otel:
|
||||
endpointAddress: "localhost:4317"
|
||||
isSecure: false
|
||||
protocol: "grpc"
|
||||
httpPipeline:
|
||||
handlers:
|
||||
- name: oauth2
|
||||
|
|
|
|||
|
|
@ -3,5 +3,5 @@ type: docs
|
|||
title: "Observe your application with Dapr"
|
||||
linkTitle: "Observability"
|
||||
weight: 80
|
||||
description: "How to monitor your application using Dapr integrations"
|
||||
description: "How to observe and gain insights into your application"
|
||||
---
|
||||
|
|
|
|||
|
|
@ -2,7 +2,7 @@
|
|||
type: docs
|
||||
title: "How-To: Set up Fluentd, Elastic search and Kibana in Kubernetes"
|
||||
linkTitle: "FluentD"
|
||||
weight: 1000
|
||||
weight: 2000
|
||||
description: "How to install Fluentd, Elastic Search, and Kibana to search logs in Kubernetes"
|
||||
---
|
||||
|
||||
|
|
|
|||
|
|
@ -2,13 +2,17 @@
|
|||
type: docs
|
||||
title: "Logs"
|
||||
linkTitle: "Logs"
|
||||
weight: 3000
|
||||
weight: 1000
|
||||
description: "Understand Dapr logging"
|
||||
---
|
||||
|
||||
Dapr produces structured logs to stdout either as a plain text or JSON formatted. By default, all Dapr processes (runtime and system services) write to console out in plain text. To enable JSON formatted logs, you need to add the `--log-as-json` command flag when running Dapr processes.
|
||||
Dapr produces structured logs to stdout either as a plain text or JSON formatted. By default, all Dapr processes (runtime and control plane services) write to console out in plain text. To enable JSON formatted logs, you need to add the `--log-as-json` command flag when running Dapr processes.
|
||||
|
||||
{{% alert title="Note" color="primary" %}}
|
||||
If you want to use a search engine such as Elastic Search or Azure Monitor to search the logs, it is strongly recommended to use JSON-formatted logs which the log collector and search engine can parse using the built-in JSON parser.
|
||||
{{% /alert %}}
|
||||
|
||||
|
||||
If you want to use a search engine such as Elastic Search or Azure Monitor to search the logs, it is recommended to use JSON-formatted logs which the log collector and search engine can parse using the built-in JSON parser.
|
||||
|
||||
## Log schema
|
||||
|
||||
|
|
|
|||
|
|
@ -2,7 +2,7 @@
|
|||
type: docs
|
||||
title: "How-To: Set-up New Relic for Dapr logging"
|
||||
linkTitle: "New Relic"
|
||||
weight: 2000
|
||||
weight: 3000
|
||||
description: "Set-up New Relic for Dapr logging"
|
||||
---
|
||||
|
||||
|
|
|
|||
|
|
@ -1,7 +1,7 @@
|
|||
---
|
||||
type: docs
|
||||
title: "Metrics"
|
||||
linkTitle: "Metrics"
|
||||
linkTitle: "View metrics"
|
||||
weight: 300
|
||||
description: "How to view Dapr metrics"
|
||||
---
|
||||
|
|
|
|||
|
|
@ -2,7 +2,7 @@
|
|||
type: docs
|
||||
title: "How-To: Set up Azure Monitor to search logs and collect metrics"
|
||||
linkTitle: "Azure Monitor"
|
||||
weight: 2000
|
||||
weight: 7000
|
||||
description: "Enable Dapr metrics and logs with Azure Monitor for Azure Kubernetes Service (AKS)"
|
||||
---
|
||||
|
||||
|
|
|
|||
|
|
@ -1,7 +1,7 @@
|
|||
---
|
||||
type: docs
|
||||
title: "How-To: Observe metrics with Grafana"
|
||||
linkTitle: "Metrics dashboards with Grafana"
|
||||
linkTitle: "Grafana dashboards"
|
||||
weight: 5000
|
||||
description: "How to view Dapr metrics in a Grafana dashboard."
|
||||
---
|
||||
|
|
|
|||
|
|
@ -1,25 +1,26 @@
|
|||
---
|
||||
type: docs
|
||||
title: "Metrics"
|
||||
linkTitle: "Metrics"
|
||||
title: "Configure metrics"
|
||||
linkTitle: "Configure metrics"
|
||||
weight: 4000
|
||||
description: "Observing Dapr metrics in Kubernetes"
|
||||
description: "Enable or disable Dapr metrics "
|
||||
---
|
||||
|
||||
Dapr exposes a [Prometheus](https://prometheus.io/) metrics endpoint that you can scrape to:
|
||||
By default, each Dapr system process emits Go runtime/process metrics and has their own [Dapr metrics](https://github.com/dapr/dapr/blob/master/docs/development/dapr-metrics.md).
|
||||
|
||||
- Gain a greater understanding of how Dapr is behaving.
|
||||
- Set up alerts for specific conditions.
|
||||
## Prometheus endpoint
|
||||
The Dapr sidecars exposes a [Prometheus](https://prometheus.io/) metrics endpoint that you can scrape to gain a greater understanding of how Dapr is behaving.
|
||||
|
||||
## Configuration
|
||||
## Configuring metrics using the CLI
|
||||
|
||||
The metrics endpoint is enabled by default. You can disable it by passing the command line argument `--enable-metrics=false` to Dapr system processes.
|
||||
The metrics application endpoint is enabled by default. You can disable it by passing the command line argument `--enable-metrics=false`.
|
||||
|
||||
The default metrics port is `9090`. You can override this by passing the command line argument `--metrics-port` to Daprd.
|
||||
|
||||
You can also disable the metrics exporter for a specific application by setting the `dapr.io/enable-metrics: "false"` annotation to your application deployment. With the metrics exporter disabled, `daprd` will not open the metrics listening port.
|
||||
## Configuring metrics in Kubernetes
|
||||
You can also enable/disable the metrics for a specific application by setting the `dapr.io/enable-metrics: "false"` annotation on your application deployment. With the metrics exporter disabled, `daprd` does not open the metrics listening port.
|
||||
|
||||
The follow example shows metrics are explicitly enabled with the port specified as "9090".
|
||||
The following Kubernetes deployment example shows how metrics are explicitly enabled with the port specified as "9090".
|
||||
|
||||
```yaml
|
||||
apiVersion: apps/v1
|
||||
|
|
@ -52,7 +53,8 @@ spec:
|
|||
imagePullPolicy: Always
|
||||
```
|
||||
|
||||
To disable the metrics collection in the Dapr side cars running in a specific namespace:
|
||||
## Configuring metrics using application configuration
|
||||
You can also enable metrics via application configuration. To disable the metrics collection in the Dapr sidecars running in a specific namespace:
|
||||
|
||||
- Use the `metrics` spec configuration.
|
||||
- Set `enabled: false` to disable the metrics in the Dapr runtime.
|
||||
|
|
@ -70,14 +72,7 @@ spec:
|
|||
enabled: false
|
||||
```
|
||||
|
||||
## Metrics
|
||||
|
||||
By default, each Dapr system process emits Go runtime/process metrics and have their own metrics:
|
||||
|
||||
- [Dapr metric list](https://github.com/dapr/dapr/blob/master/docs/development/dapr-metrics.md)
|
||||
|
||||
## References
|
||||
|
||||
* [Howto: Run Prometheus locally]({{< ref prometheus.md >}})
|
||||
* [Howto: Set up Prometheus and Grafana for metrics]({{< ref grafana.md >}})
|
||||
* [Howto: Set up Azure monitor to search logs and collect metrics for Dapr]({{< ref azure-monitor.md >}})
|
||||
|
|
|
|||
|
|
@ -3,5 +3,5 @@ type: docs
|
|||
title: "Tracing"
|
||||
linkTitle: "Tracing"
|
||||
weight: 100
|
||||
description: "How to setup your trace backend to receive application traces from Dapr"
|
||||
description: "How to setup your observability tools to receive application traces"
|
||||
---
|
||||
|
|
|
|||
|
|
@ -185,4 +185,3 @@ In your browser, go to `http://localhost:16686` and you will see the Jaeger UI.
|
|||
|
||||
## References
|
||||
- [Jaeger Getting Started](https://www.jaegertracing.io/docs/1.21/getting-started/#all-in-one)
|
||||
- [W3C distributed tracing]({{< ref w3c-tracing >}})
|
||||
|
|
@ -0,0 +1,7 @@
|
|||
---
|
||||
type: docs
|
||||
title: "Open Telemetry Collector"
|
||||
linkTitle: "Open Telemetry Collector"
|
||||
weight: 700
|
||||
description: "How to set up your observability tools to receive application traces"
|
||||
---
|
||||
|
|
@ -6,7 +6,11 @@ weight: 900
|
|||
description: "How to use Dapr to push trace events through the OpenTelemetry Collector."
|
||||
---
|
||||
|
||||
Dapr will be exporting trace in the OpenTelemetry format when OpenTelemetry is GA. In the mean time, traces can be exported using the Zipkin format. Combining with the [OpenTelemetry Collector](https://github.com/open-telemetry/opentelemetry-collector) you can still send trace to many popular tracing backends (like Azure AppInsights, AWS X-Ray, StackDriver, etc).
|
||||
{{% alert title="Note" color="primary" %}}
|
||||
Dapr directly writes traces using the OpenTelemetry (OTEL) protocol as the recommended method. For observability tools that support OTEL protocol, you do not need to use the OpenTelemetry Collector.
|
||||
|
||||
Dapr can also write traces using the Zipkin protocol. Previous to supporting the OTEL protocol, combining the Zipkin protocol with the [OpenTelemetry Collector](https://github.com/open-telemetry/opentelemetry-collector) enabled you to send traces to observability tools such as AWS X-Ray, Google Cloud Operations Suite, and Azure AppInsights. This approach remains for reference purposes only.
|
||||
{{% /alert %}}
|
||||
|
||||
|
||||
|
||||
|
|
@ -22,7 +26,6 @@ Dapr will be exporting trace in the OpenTelemetry format when OpenTelemetry is G
|
|||
|
||||
### Run OpenTelemetry Collector to push to your trace backend
|
||||
|
||||
|
||||
1. Check out the file [open-telemetry-collector-generic.yaml](/docs/open-telemetry-collector/open-telemetry-collector-generic.yaml) and replace the section marked with `<your-exporter-here>` with the correct settings for your trace exporter. Again, refer to the OpenTelemetry Collector links in the Prerequisites section to determine the correct settings.
|
||||
|
||||
2. Apply the configuration with `kubectl apply -f open-telemetry-collector-generic.yaml`.
|
||||
|
|
@ -1,17 +1,17 @@
|
|||
---
|
||||
type: docs
|
||||
title: "Configure Dapr to send distributed tracing data"
|
||||
linkTitle: "Enable Dapr tracing for your application"
|
||||
linkTitle: "Configure tracing"
|
||||
weight: 100
|
||||
description: "Configure Dapr to send distributed tracing data"
|
||||
---
|
||||
|
||||
It is recommended to run Dapr with tracing enabled for any production
|
||||
scenario. You can configure Dapr to send tracing and telemetry data
|
||||
to many backends based on your environment, whether it is running in
|
||||
to many observability tools based on your environment, whether it is running in
|
||||
the cloud or on-premises.
|
||||
|
||||
## Tracing configuration
|
||||
## Configuration
|
||||
|
||||
The `tracing` section under the `Configuration` spec contains the following properties:
|
||||
|
||||
|
|
@ -19,8 +19,11 @@ The `tracing` section under the `Configuration` spec contains the following prop
|
|||
spec:
|
||||
tracing:
|
||||
samplingRate: "1"
|
||||
otel:
|
||||
endpointAddress: "https://..."
|
||||
zipkin:
|
||||
endpointAddress: "https://..."
|
||||
|
||||
```
|
||||
|
||||
The following table lists the properties for tracing:
|
||||
|
|
@ -28,102 +31,30 @@ The following table lists the properties for tracing:
|
|||
| Property | Type | Description |
|
||||
|--------------|--------|-------------|
|
||||
| `samplingRate` | string | Set sampling rate for tracing to be enabled or disabled.
|
||||
| `zipkin.endpointAddress` | string | Set the Zipkin server address.
|
||||
| `stdout` | bool | True write more verbose information to the traces
|
||||
| `otel.endpointAddress` | string | Set the Open Telemetry (OTEL) server address.
|
||||
| `otel.isSecure` | bool | Is the connection to the endpoint address encryped.
|
||||
| `otel.protocol` | string | Set to `http` or `grpc` protocol.
|
||||
| `zipkin.endpointAddress` | string | Set the Zipkin server address. If this is used, you do not need to specify the `otel` section.
|
||||
|
||||
To enable tracing, use a configuration file (in self hosted mode) or a Kubernetes configuration object (in Kubernetes mode). For example, the following configuration object changes the sample rate to 1 (every span is sampled), and sends trace using OTEL protocol to the OTEL server at localhost:4317
|
||||
|
||||
## Zipkin in self hosted mode
|
||||
|
||||
The following steps show you how to configure Dapr to send distributed tracing data to Zipkin running as a container on your local machine and view them.
|
||||
|
||||
For self hosted mode, create a Dapr configuration file locally and reference it with the Dapr CLI.
|
||||
|
||||
1. Create the following `config.yaml` YAML file:
|
||||
|
||||
```yaml
|
||||
apiVersion: dapr.io/v1alpha1
|
||||
kind: Configuration
|
||||
metadata:
|
||||
name: zipkin
|
||||
namespace: default
|
||||
spec:
|
||||
tracing:
|
||||
samplingRate: "1"
|
||||
zipkin:
|
||||
endpointAddress: "http://localhost:9411/api/v2/spans"
|
||||
```
|
||||
|
||||
2. Launch Zipkin using Docker:
|
||||
|
||||
```bash
|
||||
docker run -d -p 9411:9411 openzipkin/zipkin
|
||||
```
|
||||
|
||||
3. Launch Dapr with the `--config` param with the path for where the `config.yaml` is saved :
|
||||
|
||||
```bash
|
||||
dapr run --app-id mynode --app-port 3000 --config ./config.yaml node app.js
|
||||
```
|
||||
|
||||
|
||||
## Zipkin in Kubernetes mode
|
||||
|
||||
The following steps show you how to configure Dapr to send distributed tracing data to Zipkin running as a container in your Kubernetes cluster, and how to view them.
|
||||
|
||||
### Setup
|
||||
|
||||
First, deploy Zipkin:
|
||||
|
||||
```bash
|
||||
kubectl create deployment zipkin --image openzipkin/zipkin
|
||||
```
|
||||
|
||||
Create a Kubernetes Service for the Zipkin pod:
|
||||
|
||||
```bash
|
||||
kubectl expose deployment zipkin --type ClusterIP --port 9411
|
||||
```
|
||||
|
||||
Next, create the following YAML file locally:
|
||||
|
||||
```yml
|
||||
```yaml
|
||||
apiVersion: dapr.io/v1alpha1
|
||||
kind: Configuration
|
||||
metadata:
|
||||
name: zipkin
|
||||
namespace: default
|
||||
name: tracing
|
||||
spec:
|
||||
tracing:
|
||||
samplingRate: "1"
|
||||
zipkin:
|
||||
endpointAddress: "http://zipkin.default.svc.cluster.local:9411/api/v2/spans"
|
||||
otel:
|
||||
endpointAddress: "localhost:4317"
|
||||
isSecure: false
|
||||
protocol: grpc
|
||||
```
|
||||
|
||||
Finally, deploy the Dapr configuration:
|
||||
## Sampling rate
|
||||
|
||||
```bash
|
||||
kubectl apply -f config.yaml
|
||||
```
|
||||
Dapr uses probabilistic sampling. The sample rate defines the probability a tracing span will be sampled and can have a value between 0 and 1 (inclusive). The default sample rate is 0.0001 (i.e. 1 in 10,000 spans is sampled).
|
||||
|
||||
In order to enable this configuration for your Dapr sidecar, add the following annotation to your pod spec template:
|
||||
|
||||
```yml
|
||||
annotations:
|
||||
dapr.io/config: "zipkin"
|
||||
```
|
||||
|
||||
That's it! Your sidecar is now configured for use with Zipkin.
|
||||
|
||||
### Viewing Tracing Data
|
||||
|
||||
To view traces, connect to the Zipkin service and open the UI:
|
||||
|
||||
```bash
|
||||
kubectl port-forward svc/zipkin 9411:9411
|
||||
```
|
||||
|
||||
On your browser, go to ```http://localhost:9411``` and you should see the Zipkin UI.
|
||||
|
||||
|
||||
|
||||
## References
|
||||
- [Zipkin for distributed tracing](https://zipkin.io/)
|
||||
Changing `samplingRate` to 0 disables tracing altogether.
|
||||
|
|
|
|||
|
|
@ -1,7 +0,0 @@
|
|||
---
|
||||
type: docs
|
||||
title: "Common Tracing Backends"
|
||||
linkTitle: "Common Tracing Backends"
|
||||
weight: 3000
|
||||
description: "How to setup various common tracing backends"
|
||||
---
|
||||
|
|
@ -2,7 +2,7 @@
|
|||
type: docs
|
||||
title: "How-To: Set up Zipkin for distributed tracing"
|
||||
linkTitle: "Zipkin"
|
||||
weight: 3000
|
||||
weight: 4000
|
||||
description: "Set up Zipkin for distributed tracing"
|
||||
type: docs
|
||||
---
|
||||
|
|
@ -108,4 +108,3 @@ In your browser, go to `http://localhost:9411` and you will see the Zipkin UI.
|
|||
|
||||
## References
|
||||
- [Zipkin for distributed tracing](https://zipkin.io/)
|
||||
- [W3C distributed tracing]({{< ref w3c-tracing >}})
|
||||
Binary file not shown.
|
Before Width: | Height: | Size: 278 KiB After Width: | Height: | Size: 189 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 157 KiB After Width: | Height: | Size: 117 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 279 KiB After Width: | Height: | Size: 215 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 28 KiB |
Loading…
Reference in New Issue