docs/howto/use-w3c-tracecontext

Trace calls across services

Contents

• Using trace context in Dapr
• How to pass a trace context
• Configuring tracing in Dapr
• Invoking Dapr with trace context
• Related Links

Using trace context in Dapr

Dapr tracing is built on OpenCensus specification that supports official W3C HTTP tracing header.

For the gRPC tracing with OpenCensus, the details are here

Dapr supports OpenCensus instrumentation of services when tracing configuration is enabled through a Dapr annotation. Once the tracing configuration is applied, you get instrumentation of traces and you can retrieve the trace correlation id from the standard W3C context headers.

You can also choose to pass the trace context explicitly, then Dapr uses the supplied trace context and propagates this all across the HTTP/gRPC call.

How to pass a trace context

Since Dapr tracing uses OpenCensus, you set the trace context using OpenCensus SDK. OpenCensus supports several different programming languages.

Language	SDK
Go	Link
Java	Link
C#	Link
C++	Link
Node.js	Link
Python	Link

Let's go through an example using the grpc app using the Go SDK.

1. Get the OpenCensus Go SDK

Prerequisites: OpenCensus Go libraries require Go 1.8 or later. For details on installation go here.

2. Import the package "go.opencensus.io/trace"

$ go get -u go.opencensus.io

3. Create trace context

When you want to pass the trace context, you are starting the trace spans. Since a distributed trace tracks the progression of a single user request as it is handled by the services and processes that make up an application, each step is called a span in the trace. Spans include metadata about the step, including the time spent in the step, called the span’s latency.

Span is the unit step in a trace. Each span has a name, latency, status and additional metadata.

The code belows shows starting a span for a cache read and ending it when done:

ctx, span := trace.StartSpan(ctx, "cache.Get")
defer span.End()

// Do work to get from cache.

The StartSpan call returns two values. If you want to propagate trace context within your calls in the same process, you can use context.Context to propagate spans. The returned span has the fields 'TraceID' and 'SpanID'. You can read more on these fields usage here

When you call the Dapr API through HTTP/gRPC, you need to pass the trace context across processes or services. Across the network, OpenCensus provides different propagation methods for different protocols. You can read about the propagation package here.

In our example, since these are gRPC calls, the OpenCensus binary propagation format is used.

4. Passing the trace context to Dapr

For gRPC calls

First import the package 'go.opencensus.io/trace/propagation'. Once imported, get the span context from the generated span (as mentioned in above step 3), and convert it to OpenCensus binary propagation format.

traceContext := span.SpanContext()
traceContextBinary := propagation.Binary(traceContext)

You can then pass the trace context through gRPC metadata through grpc-trace-bin header.

ctx = metadata.AppendToOutgoingContext(ctx, "grpc-trace-bin", string(traceContextBinary))

You can then pass this context ctx in Dapr gRPC calls as first parameter. For example InvokeService, context is passed in first parameter.

To retrieve the trace context header when the gRPC call is returned, you can pass the response header reference as gRPC call option which contains response headers:

var responseHeader metadata.MD

// Call the InvokeService with call option
// grpc.Header(&responseHeader)

client.InvokeService(ctx, &pb.InvokeServiceRequest{
		Id: "client",
		Message: &commonv1pb.InvokeRequest{
			Method:      "MyMethod",
			ContentType: "text/plain; charset=UTF-8",
			Data:        &any.Any{Value: []byte("Hello")},
		},
	},
	grpc.Header(&responseHeader))

HTTP calls

HTTP integration uses Zipkin’s B3 by default, but can be configured to use a custom propagation method by setting another propagation.HTTPFormat.

For control over HTTP client headers, redirect policy, and other settings, you create a client. In this example, net/http is used for HTTP calls.

f := &HTTPFormat{}
req, _ := http.NewRequest("GET", "http://localhost:3500/v1.0/invoke/mathService/method/api/v1/add", nil)

traceContext := span.SpanContext()
f.SpanContextToRequest(traceContext, req)

To retrieve the trace context when the HTTP request is returned, you can use :

sc, ok := f.SpanContextFromRequest(req)

See the Dapr API reference.

Configuring tracing in Dapr

To enable tracing in Dapr, you need to first configure tracing. Create adeployment config yaml e.g. appconfig.yaml with following configuration.

apiVersion: dapr.io/v1alpha1
kind: Configuration
metadata:
  name: appconfig
spec:
  tracing:
    samplingRate: "1"

In Kubernetes, you can apply the configuration as below :

kubectl apply -f appconfig.yaml

You then set the following tracing annotation in your deployment YAML. You can add the following annotaion in sample grpc app deployment yaml.

dapr.io/config: "appconfig"

Viewing traces

To view traces, you need to deploy OpenCensus supported exporters. This is independent of Dapr configuration. Read how-to-diagnose-with-tracing to set up trace exporters.

Invoking Dapr with trace context

As mentioned earlier in the section, you can create the trace context and pass it when calling Dapr, or Dapr can generate trace context and pass it back to you.

If you choose to pass the trace context explicitly, then Dapr will use the passed trace context and propagate all across the HTTP/gRPC call.

Using the grpc app in the example and putting this all together, the following steps show you how to create a Dapr client and call the Save State operation passing the trace context:

The Rest code snippet and details, refer to the grpc app.

1. Import the package

package main

import (
    pb "github.com/dapr/go-sdk/dapr"
    "go.opencensus.io/trace"
	  "go.opencensus.io/trace/propagation"
	  "google.golang.org/grpc"
	  "google.golang.org/grpc/metadata"
)

2. Create the client

  // Get the Dapr port and create a connection
  daprPort := os.Getenv("DAPR_GRPC_PORT")
  daprAddress := fmt.Sprintf("localhost:%s", daprPort)
  conn, err := grpc.Dial(daprAddress, grpc.WithInsecure())
  if err != nil {
    fmt.Println(err)
  }
  defer conn.Close()

  // Create the client
  client := pb.NewDaprClient(conn)

3. Invoke the InvokeService method With Trace Context

  // Create the Trace Context
  ctx , span := trace.StartSpan(context.Background(), "InvokeService")

  // The returned context can be used to keep propagating the newly created span in the current context.
  // In the same process, context.Context is used to propagate trace context.

  // Across the process, use the propagation format of Trace Context to propagate trace context.
  traceContext := propagation.Binary(span.SpanContext())
  ctx = metadata.NewOutgoingContext(ctx, string(traceContext))

  // Pass the trace context
  resp, err := client.InvokeService(ctx, &pb.InvokeServiceRequest{
		Id: "client",
		Message: &commonv1pb.InvokeRequest{
			Method:      "MyMethod",
			ContentType: "text/plain; charset=UTF-8",
			Data:        &any.Any{Value: []byte("Hello")},
		},
	})

That's it !. Now you can correlate the calls in your app and across services with Dapr using the same trace context.

To view traces, you can refer how-to-diagnose-with-tracing

Related Links

• Observability concepts
• Observability sample
• W3C Trace Context for distributed tracing
• How-To: Set up Application Insights for distributed tracing
• How-To: Set up Zipkin for distributed tracing
• W3C trace context specification