345 lines
16 KiB
Markdown
345 lines
16 KiB
Markdown
# Extending the OpenTelemetry .NET SDK
|
|
|
|
* [Building your own exporter](#exporter)
|
|
* [Building your own instrumentation library](#instrumentation-library)
|
|
* [Building your own processor](#processor)
|
|
* [Building your own sampler](#sampler)
|
|
* [Building your own resource detector](#resource-detector)
|
|
* [References](#references)
|
|
|
|
## Exporter
|
|
|
|
OpenTelemetry .NET SDK has provided the following built-in trace exporters:
|
|
|
|
* [Console](../../../src/OpenTelemetry.Exporter.Console/README.md)
|
|
* [InMemory](../../../src/OpenTelemetry.Exporter.InMemory/README.md)
|
|
* [Jaeger](../../../src/OpenTelemetry.Exporter.Jaeger/README.md)
|
|
* [OpenTelemetryProtocol](../../../src/OpenTelemetry.Exporter.OpenTelemetryProtocol/README.md)
|
|
* [Zipkin](../../../src/OpenTelemetry.Exporter.Zipkin/README.md)
|
|
|
|
Custom exporters can be implemented to send telemetry data to places which are
|
|
not covered by the built-in exporters:
|
|
|
|
* Exporters should derive from `OpenTelemetry.BaseExporter<Activity>` (which
|
|
belongs to the [OpenTelemetry](../../../src/OpenTelemetry/README.md) package)
|
|
and implement the `Export` method.
|
|
* Exporters can optionally implement the `OnForceFlush` and `OnShutdown` method.
|
|
* Depending on user's choice and load on the application, `Export` may get
|
|
called with one or more activities.
|
|
* Exporters will only receive sampled-in and ended activities.
|
|
* Exporters should not throw exceptions from `Export`, `OnForceFlush` and
|
|
`OnShutdown`.
|
|
* Exporters should not modify activities they receive (the same activity may be
|
|
exported again by different exporter).
|
|
* Exporters are responsible for any retry logic needed by the scenario. The SDK
|
|
does not implement any retry logic.
|
|
* Exporters should avoid generating telemetry and causing live-loop, this can be
|
|
done via `OpenTelemetry.SuppressInstrumentationScope`.
|
|
* Exporters should use `Activity.TagObjects` collection instead of
|
|
`Activity.Tags` to obtain the full set of attributes (tags).
|
|
* Exporters should use `ParentProvider.GetResource()` to get the `Resource`
|
|
associated with the provider.
|
|
|
|
```csharp
|
|
class MyExporter : BaseExporter<Activity>
|
|
{
|
|
public override ExportResult Export(in Batch<Activity> batch)
|
|
{
|
|
using var scope = SuppressInstrumentationScope.Begin();
|
|
|
|
foreach (var activity in batch)
|
|
{
|
|
Console.WriteLine($"Export: {activity.DisplayName}");
|
|
}
|
|
|
|
return ExportResult.Success;
|
|
}
|
|
}
|
|
```
|
|
|
|
A demo exporter which simply writes activity name to the console is shown
|
|
[here](./MyExporter.cs).
|
|
|
|
Apart from the exporter itself, you should also provide extension methods as
|
|
shown [here](./MyExporterExtensions.cs). This allows users to add the Exporter
|
|
to the `TracerProvider` as shown in the example [here](./Program.cs).
|
|
|
|
### Exporting Activity Status
|
|
|
|
[DiagnosticSource](https://www.nuget.org/packages/system.diagnostics.diagnosticsource)
|
|
package did not originally have a dedicated field for storing
|
|
[Status](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/api.md#set-status),
|
|
and hence, users were encouraged to follow the convention of storing status
|
|
using tags "otel.status_code" and "otel.status_description".
|
|
[DiagnosticSource](https://www.nuget.org/packages/system.diagnostics.diagnosticsource)
|
|
version 6.0.0 added `Status` and `StatusDescription` to `Activity` class.
|
|
Exporters which support reading status from `Activity` directly should fall back
|
|
to retrieving status from the tags described above, to maintain backward
|
|
compatibility.
|
|
[ConsoleActivityExporter](../../../src/OpenTelemetry.Exporter.Console/ConsoleActivityExporter.cs)
|
|
may be used as a reference.
|
|
|
|
## Instrumentation Library
|
|
|
|
The [inspiration of the OpenTelemetry
|
|
project](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/overview.md#instrumentation-libraries)
|
|
is to make every library observable out of the box by having them call
|
|
OpenTelemetry API directly. However, many libraries will not have such
|
|
integration, and as such there is a need for a separate library which would
|
|
inject such calls, using mechanisms such as wrapping interfaces, subscribing to
|
|
library-specific callbacks, or translating existing telemetry into OpenTelemetry
|
|
model.
|
|
|
|
A library which enables instrumentation for another library is called
|
|
[Instrumentation
|
|
Library](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/glossary.md#instrumentation-library)
|
|
and the library it instruments is called the [Instrumented
|
|
Library](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/glossary.md#instrumented-library).
|
|
If a given library has built-in instrumentation with OpenTelemetry, then
|
|
instrumented library and instrumentation library will be the same.
|
|
|
|
The [OpenTelemetry .NET Github repo](../../../README.md#getting-started) ships
|
|
the following instrumentation libraries. The individual docs for them describes
|
|
the library they instrument, and steps for enabling them.
|
|
|
|
* [ASP.NET
|
|
Core](../../../src/OpenTelemetry.Instrumentation.AspNetCore/README.md)
|
|
* [gRPC
|
|
client](../../../src/OpenTelemetry.Instrumentation.GrpcNetClient/README.md)
|
|
* [HTTP clients](../../../src/OpenTelemetry.Instrumentation.Http/README.md)
|
|
* [SQL client](../../../src/OpenTelemetry.Instrumentation.SqlClient/README.md)
|
|
|
|
More community contributed instrumentations are available in [OpenTelemetry .NET
|
|
Contrib](https://github.com/open-telemetry/opentelemetry-dotnet-contrib/tree/main/src).
|
|
If you are writing an instrumentation library yourself, use the following
|
|
guidelines.
|
|
|
|
### Writing own instrumentation library
|
|
|
|
This section describes the steps required to write your own instrumentation
|
|
library.
|
|
|
|
*If you are writing a new library or modifying an existing library, the
|
|
recommendation is to use [ActivitySource API/OpenTelemetry
|
|
API](../../../src/OpenTelemetry.Api/README.md#introduction-to-opentelemetry-net-tracing-api)
|
|
to instrument it and emit activity/span. If a library is instrumented using
|
|
ActivitySource API, then there is no need of writing a separate instrumentation
|
|
library, as instrumented and instrumentation library become same in this case.
|
|
For applications to collect traces from this library, all that is needed is to
|
|
enable the ActivitySource for the library using `AddSource` method of the
|
|
`TracerProviderBuilder`. The following section is applicable only if you are
|
|
writing an instrumentation library for an instrumented library which you cannot
|
|
modify to emit activities directly.*
|
|
|
|
Writing an instrumentation library typically involves 3 steps.
|
|
|
|
1. First step involves "hijacking" into the target library. The exact mechanism
|
|
of this depends on the target library itself. For example,
|
|
System.Data.SqlClient for .NET Framework, which publishes events using
|
|
`EventSource`. The [SqlClient instrumentation
|
|
library](../../../src/OpenTelemetry.Instrumentation.SqlClient/Implementation/SqlEventSourceListener.netfx.cs),
|
|
in this case subscribes to the `EventSource` callbacks.
|
|
|
|
2. Second step is to emit activities using the [ActivitySource
|
|
API](../../../src/OpenTelemetry.Api/README.md#introduction-to-opentelemetry-net-tracing-api).
|
|
In this step, the instrumentation library emits activities *on behalf of* the
|
|
target instrumented library. Irrespective of the actual mechanism used in
|
|
first step, this should be uniform across all instrumentation libraries. The
|
|
`ActivitySource` must be created using the name and version of the
|
|
instrumentation library (eg: "OpenTelemetry.Instrumentation.Http") and *not*
|
|
the instrumented library (eg: "System.Net.Http")
|
|
1. [Context
|
|
Propagation](../../../src/OpenTelemetry.Api/README.md#context-propagation):
|
|
If your library initiates out of process requests or accepts them, the
|
|
library needs to [inject the
|
|
`PropagationContext`](../../../examples/MicroserviceExample/Utils/Messaging/MessageSender.cs)
|
|
to outgoing requests and [extract the
|
|
context](../../../examples/MicroserviceExample/Utils/Messaging/MessageReceiver.cs)
|
|
and hydrate the Activity/Baggage upon receiving incoming requests. This is
|
|
only required if you're using your own protocol to communicate over the
|
|
wire. (i.e. If you're using an already instrumented HttpClient or
|
|
GrpcClient, this is already provided to you and **do not require**
|
|
injecting/extracting `PropagationContext` explicitly again.)
|
|
|
|
3. Third step is an optional step, and involves providing extension methods on
|
|
`TracerProviderBuilder`, to enable the instrumentation. This is optional, and
|
|
the below guidance must be followed:
|
|
|
|
1. If the instrumentation library requires state management tied to that of
|
|
`TracerProvider`, then it must register itself with the provider with the
|
|
`AddInstrumentation` method on the `TracerProviderBuilder`. This causes
|
|
the instrumentation to be created and disposed along with
|
|
`TracerProvider`. If the above is required, then it must provide an
|
|
extension method on `TracerProviderBuilder`. Inside this extension
|
|
method, it should call the `AddInstrumentation` method, and `AddSource`
|
|
method to enable its ActivitySource for the provider. An example
|
|
instrumentation using this approach is [SqlClient
|
|
instrumentation](../../../src/OpenTelemetry.Instrumentation.SqlClient/TracerProviderBuilderExtensions.cs)
|
|
|
|
2. If the instrumentation library does not requires any state management
|
|
tied to that of `TracerProvider`, then providing `TracerProviderBuilder`
|
|
extension method is optional. If provided, then it must call `AddSource`
|
|
to enable its ActivitySource for the provider.
|
|
|
|
3. If instrumentation library does not require state management, and is not
|
|
providing extension method, then the name of the `ActivitySource` used by
|
|
the instrumented library must be documented so that end users can enable
|
|
it using `AddSource` method on `TracerProviderBuilder`.
|
|
|
|
### Special case : Instrumentation for libraries producing legacy Activity
|
|
|
|
There is a special case for libraries which are already instrumented to produce
|
|
[Activity](https://github.com/dotnet/runtime/blob/master/src/libraries/System.Diagnostics.DiagnosticSource/src/ActivityUserGuide.md),
|
|
but using the
|
|
[DiagnosticSource](https://github.com/dotnet/runtime/blob/master/src/libraries/System.Diagnostics.DiagnosticSource/src/DiagnosticSourceUsersGuide.md)
|
|
method. These are referred to as "legacy Activity" in this repo. These libraries
|
|
already create activities but they do so by using the `Activity` constructor
|
|
directly, rather than using `ActivitySource.StartActivity` method. These
|
|
activities does not by default runs through the sampler, and will have their
|
|
`Kind` set to internal and they'll have empty ActivitySource name associated
|
|
with it.
|
|
|
|
Some common examples of such libraries include [ASP.NET
|
|
Core](../../../src/OpenTelemetry.Instrumentation.AspNetCore/README.md), [HTTP
|
|
client .NET Core](../../../src/OpenTelemetry.Instrumentation.Http/README.md) .
|
|
Instrumentation libraries for these are already provided in this repo. The
|
|
[OpenTelemetry .NET
|
|
Contrib](https://github.com/open-telemetry/opentelemetry-dotnet-contrib)
|
|
repository also has instrumentations for libraries like
|
|
[ElasticSearchClient](https://github.com/open-telemetry/opentelemetry-dotnet-contrib/tree/main/src/OpenTelemetry.Contrib.Instrumentation.ElasticsearchClient)
|
|
etc. which fall in this category.
|
|
|
|
If you are writing instrumentation for such library, it is recommended to refer
|
|
to one of the above as a reference.
|
|
|
|
## Processor
|
|
|
|
OpenTelemetry .NET SDK has provided the following built-in processors:
|
|
|
|
* [BatchExportProcessor<T>](../../../src/OpenTelemetry/BatchExportProcessor.cs)
|
|
* [CompositeProcessor<T>](../../../src/OpenTelemetry/CompositeProcessor.cs)
|
|
* [SimpleExportProcessor<T>](../../../src/OpenTelemetry/SimpleExportProcessor.cs)
|
|
|
|
Custom processors can be implemented to cover more scenarios:
|
|
|
|
* Processors should inherit from `OpenTelemetry.BaseProcessor<Activity>` (which
|
|
belongs to the [OpenTelemetry](../../../src/OpenTelemetry/README.md) package),
|
|
and implement the `OnStart` and `OnEnd` methods.
|
|
* Processors can optionally implement the `OnForceFlush` and `OnShutdown`
|
|
methods. `OnForceFlush` should be thread safe.
|
|
* Processors should not throw exceptions from `OnStart`, `OnEnd`, `OnForceFlush`
|
|
and `OnShutdown`.
|
|
* `OnStart` and `OnEnd` should be thread safe, and should not block or take long
|
|
time, since they will be called on critical code path.
|
|
|
|
```csharp
|
|
class MyProcessor : BaseProcessor<Activity>
|
|
{
|
|
public override void OnStart(Activity activity)
|
|
{
|
|
Console.WriteLine($"OnStart: {activity.DisplayName}");
|
|
}
|
|
|
|
public override void OnEnd(Activity activity)
|
|
{
|
|
Console.WriteLine($"OnEnd: {activity.DisplayName}");
|
|
}
|
|
}
|
|
```
|
|
|
|
A demo processor is shown [here](./MyProcessor.cs).
|
|
|
|
### Enriching Processor
|
|
|
|
A common use case of writing custom processor is to enrich activities with
|
|
additional tags. An example of such an "EnrichingProcessor" is shown
|
|
[here](./MyEnrichingProcessor.cs). Such processors must be added *before* the
|
|
exporters.
|
|
|
|
This processor also shows how to enrich `Activity` with additional tags from the
|
|
`Baggage`.
|
|
|
|
Many [instrumentation libraries](#instrumentation-library) shipped from this
|
|
repo provides a built-in `Enrich` option, which may also be used to enrich
|
|
activities. Instrumentation library provided approach may offer additional
|
|
capabilities such as offering easy access to more context (library specific).
|
|
|
|
### Filtering Processor
|
|
|
|
Another common use case of writing custom processor is to filter Activities from
|
|
being exported. Such a "FilteringProcessor" can be written as a wrapper around
|
|
an underlying processor. An example "FilteringProcessor" is shown
|
|
[here](./MyFilteringProcessor.cs).
|
|
|
|
When using such a filtering processor, instead of using extension method to
|
|
register the exporter, they must be registered manually as shown below:
|
|
|
|
```csharp
|
|
using var tracerProvider = Sdk.CreateTracerProviderBuilder()
|
|
.SetSampler(new MySampler())
|
|
.AddSource("OTel.Demo")
|
|
.AddProcessor(new MyFilteringProcessor(
|
|
new SimpleActivityExportProcessor(new MyExporter("ExporterX")),
|
|
(act) => true))
|
|
.Build();
|
|
```
|
|
|
|
Most [instrumentation libraries](#instrumentation-library) shipped from this
|
|
repo provides a built-in `Filter` option to achieve the same effect. In such
|
|
cases, it is recommended to use that option as it offers higher performance.
|
|
|
|
## Sampler
|
|
|
|
OpenTelemetry .NET SDK has provided the following built-in samplers:
|
|
|
|
* [AlwaysOffSampler](../../../src/OpenTelemetry/Trace/AlwaysOffSampler.cs)
|
|
* [AlwaysOnSampler](../../../src/OpenTelemetry/Trace/AlwaysOnSampler.cs)
|
|
* [ParentBasedSampler](../../../src/OpenTelemetry/Trace/ParentBasedSampler.cs)
|
|
* [TraceIdRatioBasedSampler](../../../src/OpenTelemetry/Trace/TraceIdRatioBasedSampler.cs)
|
|
|
|
Custom samplers can be implemented to cover more scenarios:
|
|
|
|
* Samplers should inherit from `OpenTelemetry.Trace.Sampler` (which belongs to
|
|
the [OpenTelemetry](../../../src/OpenTelemetry/README.md) package), and
|
|
implement the `ShouldSample` method.
|
|
* `ShouldSample` should be thread safe, and should not block or take long time,
|
|
since it will be called on critical code path.
|
|
|
|
```csharp
|
|
class MySampler : Sampler
|
|
{
|
|
public override SamplingResult ShouldSample(in SamplingParameters samplingParameters)
|
|
{
|
|
return new SamplingResult(SamplingDecision.RecordAndSampled);
|
|
}
|
|
}
|
|
```
|
|
|
|
A demo sampler is shown [here](./MySampler.cs).
|
|
|
|
## Resource Detector
|
|
|
|
OpenTelemetry .NET SDK provides a resource detector for detecting resource
|
|
information from the `OTEL_RESOURCE_ATTRIBUTES` and `OTEL_SERVICE_NAME`
|
|
environment variables.
|
|
|
|
Custom resource detectors can be implemented:
|
|
|
|
* ResourceDetectors should inherit from
|
|
`OpenTelemetry.Resources.IResourceDetector`, (which belongs to the
|
|
[OpenTelemetry](../../../src/OpenTelemetry/README.md) package), and implement
|
|
the `Detect` method.
|
|
|
|
A demo ResourceDetector is shown [here](./MyResourceDetector.cs).
|
|
|
|
## References
|
|
|
|
* [Exporter
|
|
specification](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/sdk.md#span-exporter)
|
|
* [Processor
|
|
specification](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/sdk.md#span-processor)
|
|
* [Resource
|
|
specification](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/resource/sdk.md)
|
|
* [Sampler
|
|
specification](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/sdk.md#sampler)
|