opentelemetry-java-instrumentation/dd-trace

Datadog OpenTracing Tracer

Motivations

The Datadog Tracer is an OpenTracing compatible tracer. It provides all resources needed to instrument your code and report each operation and each trace directly to a Datadog APM platform.

OpenTracing uses the concept of spans and traces. A span is timed operation representing a bunch of work executed. Spans can be linked together. A trace is a collection of spans, related to the same top action/operation.

Let's see an example.

For instance, a client requesting a resource through an HTTP endpoint. Look at the following workflow.

TRACE:

   client ---> HTTP Endpoint ---> DB Query/Result ---> Custom processing ---> client
  
   SPAN 1 (Root Span) ................................................... (end)   - 200ms
   |--------SPAN 2 (Child of 1).................................(end)             - 100ms
            |-----------SPAN 3 (Child of 2).(end)                                 - 50 ms
            |----------------------------------------SPAN 4 (Child of 2)..(end)   - 50 ms

OpenTracing provides a way for measuring the time consumed for each operation. As just described, the tracer produces a trace composed of 4 spans, each representing a specific action:

1. Span1 is the time from doing the request to getting the response.
2. Span2 is the Span1's first child, representing the amount of time to understand the query, and perform the query on the DB.
3. Span3 is Span1's grandchild, represents the DB time used to retrieve the data
4. Span4 is a child of Span2 and follows Span3. It represents a business/legacy operation.

This is a very simple example of how works OpenTracing. For more information, see http://docs.datadoghq.com/tracing/terminology/ or http://opentracing.io/.

How to instrument your application?

In order to start to instrument your application, you need to:

1. Configure the Datadog Tracer
2. Choose one of the 3 ways to instrument an application:
   1. Use dd-java-agent for supported frawemorks
   2. Use the OpenTracing API in your code
   3. Use annotations

Datadog Tracer configuration

The DDTracer is configured using a YAML file named dd-trace.yaml following the structure below.

By default, the DDTracer tries to reach a local Datadog Agent, but you can change the settings and use a different location. In order to do that, please, refer you to the latest configuration: dd-trace.yaml

Make sure that file is present in your classpath.

# Service name used if none is provided in the app
defaultServiceName: unnamed-java-app

# The writer to use.
# Could be: LoggingWritter or DDAgentWriter (default)
writer:
  # LoggingWriter: Spans are logged using the application configuration
  # DDAgentWriter: Spans are forwarding to a Datadog Agent
  #  - Param 'host': the hostname where the DD Agent running (default: localhost)
  #  - Param 'port': the port to reach the DD Agent (default: 8126)
  type: DDAgentWriter
  host: localhost
  port: 8126

# The sampler to use.
# Could be: AllSampler (default) or RateSampler
sampler:
  # AllSampler: all spans are reported to the writer
  # RateSample: only a portion of spans are reported to the writer
  #  - Param 'rate': the portion of spans to keep
  type: AllSampler

Use the Datadog Java agent for well-known framework

Datadog uses instrumentation contributed by the community to instrument many frameworks: SpringBoot, JDBC, Mongo, JMS, Tomcat, etc. By using dd-java-agent, you just need to follow few steps in order to get traces.

Get the latest version of the Datadog Java agent (Do not forget to replace the version ${version} by the appropriate one).

version=0.1.1
curl -OL http://central.maven.org/maven2/com/datadoghq/dd-java-agent/${version}/dd-java-agent-${version}.jar 

Then, attach the Java agent to your JVM using th javaagent option.

java -javaagent:/path/to/dd-java-agent-${version}.jar ...

If you have a local Datadog agent running on your host, traces are visible in your Datadog account.

You can choose which framework you want to instrument, or sending traces to a remote Datadog agent by configuring the Datadog Java Agent YAML file. Check the dedicated project for the full documentation: dd-java-agent

Custom instrumentations using OpenTracing API

Rather than referencing classes directly from dd-trace (other than registering DDTracer), we strongly suggest using the OpenTracing API. Additional documentation on the api is also available.

Let's look at a simple example.

class InstrumentedClass {
    
    void method0() {
        // Retrieve the tracer using the resolver provided
        // Make sure you have :
        //    1. added the agent to the jvm (-javaagent;/path/to/agent.jar)
        //    2. a dd-trace.yaml file in your resources directory
        Tracer tracer = io.opentracing.util.GlobalTracer.get();
        
        Span span = tracer.buildSpan("operation-name").startActive();
        new io.opentracing.tag.StringTag("service-name").set(span, "new-service-name"); 
        
        
        //Do some thing here ...
        Thread.sleep(1_000);
        
        // Close the span, the trace will automatically reported to the writer configured
        span.finish();   
    }	
	
}

The method above is now instrumented. As you can see, the tracer is retrieved from a global registry, called GlobalTracer.

The last thing you have to do is providing a configured tracer. This can be easily done by using the TracerFactory or manually in the bootstrap method (like the main).

public class Application {

    public static void main(String[] args) {
	
        // Init the tracer from the configuration file      
        Tracer tracer = DDTracerFactory.createFromConfigurationFile();
        io.opentracing.util.GlobalTracer.register(tracer);
        
        // OR
        // Init the tracer from the API
        Writer writer = new com.datadoghq.trace.writer.DDAgentWriter();
        Sampler sampler = new com.datadoghq.trace.sampling.AllSampler();
        Tracer tracer = new com.datadoghq.trace.DDTracer(writer, sampler);
        io.opentracing.util.GlobalTracer.register(tracer);
        
        // ...
    }
}

DDTracerFactory looks for a dd-trace.yaml file in the classpath.

Finally, do not forget to add the corresponding dependencies to your project.

Maven:

        <!-- OpenTracing API -->
        <dependency>
            <groupId>io.opentracing</groupId>
            <artifactId>opentracing-api</artifactId>
            <version>${opentracing.version}</version>
        </dependency>
        
        <!-- Datadog Tracer (only needed if you do not use dd-java-agent) -->
        <dependency>
            <groupId>com.datadoghq</groupId>
            <artifactId>dd-trace</artifactId>
            <version>${dd-trace-java.version}</version>
        </dependency>

Gradle:

        compile group: 'io.opentracing', name: 'opentracing-api', version: "${opentracing.version}"
        compile group: 'com.datadoghq', name: 'dd-trace', version: "${dd-trace-java.version}"

Adding Trace annotations to your methods

An easy way to improve visibility to your application is by adding the @Trace annotation on the methods you want to instrument. This is equivelent to the method0 example from the api section.

class InstrumentedClass {

    @Trace(operationName = "operation-name-1")
    void method1() {

        //Do some thing here ...
        Thread.sleep(1_000);
    }	
    
    @Trace(operationName = "operation-name-2")
    void method2() {
        
        // You can get the current span and add tag as follow
        Span current = io.opentracing.util.GlobalTracer.get().activeSpan();
        new io.opentracing.tag.StringTag("service-name").set(current, "new-service-name");

        //Do some thing here ...
        Thread.sleep(1_000);
    }	
}

In order to use annotations, the only required dependency is dd-trace-annotations. Maven:

        <!-- Datadog annotations -->
        <dependency>
            <groupId>com.datadoghq</groupId>
            <artifactId>dd-trace-annotations</artifactId>
            <version>${dd-trace-java.version}</version>
        </dependency>

Gradle:

        compile group: 'com.datadoghq', name: 'dd-trace-annotations', version: "${dd-trace-java.version}"

The annotations are resolved at the runtime by the Datadog Java agent. If you want to use the annotations, so you must run the Datadog Java Agent.

To run the agent, please refer to the Datadog Java agent documentation: dd-java-agent

Other useful resources

Before instrumenting your own project you might want to review the provided examples:

• Dropwizard/MongoDB & Cross process client calls
• Springboot & H2 over JDBC

Other links that you might want to read:

• Improve your APM experience for apps running on docker by enabling the Docker Agent
• Datadog's APM Terminology
• FAQ