# Extending the OpenTelemetry .NET SDK * [Building your own exporter](#exporter) * [Building your own instrumentation library](#instrumentation-library) * [Building your own processor](#processor) * [Building your own sampler](#sampler) * [Building your own resource detector](#resource-detector) * [References](#references) ## Exporter OpenTelemetry .NET SDK has provided the following built-in trace exporters: * [Console](../../../src/OpenTelemetry.Exporter.Console/README.md) * [InMemory](../../../src/OpenTelemetry.Exporter.InMemory/README.md) * [Jaeger](../../../src/OpenTelemetry.Exporter.Jaeger/README.md) * [OpenTelemetryProtocol](../../../src/OpenTelemetry.Exporter.OpenTelemetryProtocol/README.md) * [Zipkin](../../../src/OpenTelemetry.Exporter.Zipkin/README.md) Custom exporters can be implemented to send telemetry data to places which are not covered by the built-in exporters: * Exporters should derive from `OpenTelemetry.BaseExporter` (which belongs to the [OpenTelemetry](../../../src/OpenTelemetry/README.md) package) and implement the `Export` method. * Exporters can optionally implement the `OnForceFlush` and `OnShutdown` method. * Depending on user's choice and load on the application, `Export` may get called with one or more activities. * Exporters will only receive sampled-in and ended activities. * Exporters should not throw exceptions from `Export`, `OnForceFlush` and `OnShutdown`. * Exporters should not modify activities they receive (the same activity may be exported again by different exporter). * Exporters are responsible for any retry logic needed by the scenario. The SDK does not implement any retry logic. * Exporters should avoid generating telemetry and causing live-loop, this can be done via `OpenTelemetry.SuppressInstrumentationScope`. * Exporters should use `Activity.TagObjects` collection instead of `Activity.Tags` to obtain the full set of attributes (tags). * Exporters should use `ParentProvider.GetResource()` to get the `Resource` associated with the provider. ```csharp class MyExporter : BaseExporter { public override ExportResult Export(in Batch batch) { using var scope = SuppressInstrumentationScope.Begin(); foreach (var activity in batch) { Console.WriteLine($"Export: {activity.DisplayName}"); } return ExportResult.Success; } } ``` A demo exporter which simply writes activity name to the console is shown [here](./MyExporter.cs). Apart from the exporter itself, you should also provide extension methods as shown [here](./MyExporterExtensions.cs). This allows users to add the Exporter to the `TracerProvider` as shown in the example [here](./Program.cs). ### Exporting Activity Status [DiagnosticSource](https://www.nuget.org/packages/system.diagnostics.diagnosticsource) package did not originally have a dedicated field for storing [Status](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/api.md#set-status), and hence, users were encouraged to follow the convention of storing status using tags "otel.status_code" and "otel.status_description". [DiagnosticSource](https://www.nuget.org/packages/system.diagnostics.diagnosticsource) version 6.0.0 added `Status` and `StatusDescription` to `Activity` class. Exporters which support reading status from `Activity` directly should fall back to retrieving status from the tags described above, to maintain backward compatibility. [ConsoleActivityExporter](../../../src/OpenTelemetry.Exporter.Console/ConsoleActivityExporter.cs) may be used as a reference. ## Instrumentation Library The [inspiration of the OpenTelemetry project](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/overview.md#instrumentation-libraries) is to make every library observable out of the box by having them call OpenTelemetry API directly. However, many libraries will not have such integration, and as such there is a need for a separate library which would inject such calls, using mechanisms such as wrapping interfaces, subscribing to library-specific callbacks, or translating existing telemetry into OpenTelemetry model. A library which enables instrumentation for another library is called [Instrumentation Library](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/glossary.md#instrumentation-library) and the library it instruments is called the [Instrumented Library](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/glossary.md#instrumented-library). If a given library has built-in instrumentation with OpenTelemetry, then instrumented library and instrumentation library will be the same. The [OpenTelemetry .NET Github repo](../../../README.md#getting-started) ships the following instrumentation libraries. The individual docs for them describes the library they instrument, and steps for enabling them. * [ASP.NET Core](../../../src/OpenTelemetry.Instrumentation.AspNetCore/README.md) * [gRPC client](../../../src/OpenTelemetry.Instrumentation.GrpcNetClient/README.md) * [HTTP clients](../../../src/OpenTelemetry.Instrumentation.Http/README.md) * [SQL client](../../../src/OpenTelemetry.Instrumentation.SqlClient/README.md) More community contributed instrumentations are available in [OpenTelemetry .NET Contrib](https://github.com/open-telemetry/opentelemetry-dotnet-contrib/tree/main/src). If you are writing an instrumentation library yourself, use the following guidelines. ### Writing own instrumentation library This section describes the steps required to write your own instrumentation library. *If you are writing a new library or modifying an existing library, the recommendation is to use [ActivitySource API/OpenTelemetry API](../../../src/OpenTelemetry.Api/README.md#introduction-to-opentelemetry-net-tracing-api) to instrument it and emit activity/span. If a library is instrumented using ActivitySource API, then there is no need of writing a separate instrumentation library, as instrumented and instrumentation library become same in this case. For applications to collect traces from this library, all that is needed is to enable the ActivitySource for the library using `AddSource` method of the `TracerProviderBuilder`. The following section is applicable only if you are writing an instrumentation library for an instrumented library which you cannot modify to emit activities directly.* Writing an instrumentation library typically involves 3 steps. 1. First step involves "hijacking" into the target library. The exact mechanism of this depends on the target library itself. For example, System.Data.SqlClient for .NET Framework, which publishes events using `EventSource`. The [SqlClient instrumentation library](../../../src/OpenTelemetry.Instrumentation.SqlClient/Implementation/SqlEventSourceListener.netfx.cs), in this case subscribes to the `EventSource` callbacks. 2. Second step is to emit activities using the [ActivitySource API](../../../src/OpenTelemetry.Api/README.md#introduction-to-opentelemetry-net-tracing-api). In this step, the instrumentation library emits activities *on behalf of* the target instrumented library. Irrespective of the actual mechanism used in first step, this should be uniform across all instrumentation libraries. The `ActivitySource` must be created using the name and version of the instrumentation library (eg: "OpenTelemetry.Instrumentation.Http") and *not* the instrumented library (eg: "System.Net.Http") 1. [Context Propagation](../../../src/OpenTelemetry.Api/README.md#context-propagation): If your library initiates out of process requests or accepts them, the library needs to [inject the `PropagationContext`](../../../examples/MicroserviceExample/Utils/Messaging/MessageSender.cs) to outgoing requests and [extract the context](../../../examples/MicroserviceExample/Utils/Messaging/MessageReceiver.cs) and hydrate the Activity/Baggage upon receiving incoming requests. This is only required if you're using your own protocol to communicate over the wire. (i.e. If you're using an already instrumented HttpClient or GrpcClient, this is already provided to you and **do not require** injecting/extracting `PropagationContext` explicitly again.) 3. Third step is an optional step, and involves providing extension methods on `TracerProviderBuilder`, to enable the instrumentation. This is optional, and the below guidance must be followed: 1. If the instrumentation library requires state management tied to that of `TracerProvider`, then it must register itself with the provider with the `AddInstrumentation` method on the `TracerProviderBuilder`. This causes the instrumentation to be created and disposed along with `TracerProvider`. If the above is required, then it must provide an extension method on `TracerProviderBuilder`. Inside this extension method, it should call the `AddInstrumentation` method, and `AddSource` method to enable its ActivitySource for the provider. An example instrumentation using this approach is [SqlClient instrumentation](../../../src/OpenTelemetry.Instrumentation.SqlClient/TracerProviderBuilderExtensions.cs) 2. If the instrumentation library does not requires any state management tied to that of `TracerProvider`, then providing `TracerProviderBuilder` extension method is optional. If provided, then it must call `AddSource` to enable its ActivitySource for the provider. 3. If instrumentation library does not require state management, and is not providing extension method, then the name of the `ActivitySource` used by the instrumented library must be documented so that end users can enable it using `AddSource` method on `TracerProviderBuilder`. ### Special case : Instrumentation for libraries producing legacy Activity There is a special case for libraries which are already instrumented to produce [Activity](https://github.com/dotnet/runtime/blob/master/src/libraries/System.Diagnostics.DiagnosticSource/src/ActivityUserGuide.md), but using the [DiagnosticSource](https://github.com/dotnet/runtime/blob/master/src/libraries/System.Diagnostics.DiagnosticSource/src/DiagnosticSourceUsersGuide.md) method. These are referred to as "legacy Activity" in this repo. These libraries already create activities but they do so by using the `Activity` constructor directly, rather than using `ActivitySource.StartActivity` method. These activities does not by default runs through the sampler, and will have their `Kind` set to internal and they'll have empty ActivitySource name associated with it. Some common examples of such libraries include [ASP.NET Core](../../../src/OpenTelemetry.Instrumentation.AspNetCore/README.md), [HTTP client .NET Core](../../../src/OpenTelemetry.Instrumentation.Http/README.md) . Instrumentation libraries for these are already provided in this repo. The [OpenTelemetry .NET Contrib](https://github.com/open-telemetry/opentelemetry-dotnet-contrib) repository also has instrumentations for libraries like [ElasticSearchClient](https://github.com/open-telemetry/opentelemetry-dotnet-contrib/tree/main/src/OpenTelemetry.Contrib.Instrumentation.ElasticsearchClient) etc. which fall in this category. If you are writing instrumentation for such library, it is recommended to refer to one of the above as a reference. ## Processor OpenTelemetry .NET SDK has provided the following built-in processors: * [BatchExportProcessor<T>](../../../src/OpenTelemetry/BatchExportProcessor.cs) * [CompositeProcessor<T>](../../../src/OpenTelemetry/CompositeProcessor.cs) * [SimpleExportProcessor<T>](../../../src/OpenTelemetry/SimpleExportProcessor.cs) Custom processors can be implemented to cover more scenarios: * Processors should inherit from `OpenTelemetry.BaseProcessor` (which belongs to the [OpenTelemetry](../../../src/OpenTelemetry/README.md) package), and implement the `OnStart` and `OnEnd` methods. * Processors can optionally implement the `OnForceFlush` and `OnShutdown` methods. `OnForceFlush` should be thread safe. * Processors should not throw exceptions from `OnStart`, `OnEnd`, `OnForceFlush` and `OnShutdown`. * `OnStart` and `OnEnd` should be thread safe, and should not block or take long time, since they will be called on critical code path. ```csharp class MyProcessor : BaseProcessor { public override void OnStart(Activity activity) { Console.WriteLine($"OnStart: {activity.DisplayName}"); } public override void OnEnd(Activity activity) { Console.WriteLine($"OnEnd: {activity.DisplayName}"); } } ``` A demo processor is shown [here](./MyProcessor.cs). ### Enriching Processor A common use case of writing custom processor is to enrich activities with additional tags. An example of such an "EnrichingProcessor" is shown [here](./MyEnrichingProcessor.cs). Such processors must be added *before* the exporters. This processor also shows how to enrich `Activity` with additional tags from the `Baggage`. Many [instrumentation libraries](#instrumentation-library) shipped from this repo provides a built-in `Enrich` option, which may also be used to enrich activities. Instrumentation library provided approach may offer additional capabilities such as offering easy access to more context (library specific). ### Filtering Processor Another common use case of writing custom processor is to filter Activities from being exported. Such a "FilteringProcessor" can be written as a wrapper around an underlying processor. An example "FilteringProcessor" is shown [here](./MyFilteringProcessor.cs). When using such a filtering processor, instead of using extension method to register the exporter, they must be registered manually as shown below: ```csharp using var tracerProvider = Sdk.CreateTracerProviderBuilder() .SetSampler(new MySampler()) .AddSource("OTel.Demo") .AddProcessor(new MyFilteringProcessor( new SimpleActivityExportProcessor(new MyExporter("ExporterX")), (act) => true)) .Build(); ``` Most [instrumentation libraries](#instrumentation-library) shipped from this repo provides a built-in `Filter` option to achieve the same effect. In such cases, it is recommended to use that option as it offers higher performance. ## Sampler OpenTelemetry .NET SDK has provided the following built-in samplers: * [AlwaysOffSampler](../../../src/OpenTelemetry/Trace/AlwaysOffSampler.cs) * [AlwaysOnSampler](../../../src/OpenTelemetry/Trace/AlwaysOnSampler.cs) * [ParentBasedSampler](../../../src/OpenTelemetry/Trace/ParentBasedSampler.cs) * [TraceIdRatioBasedSampler](../../../src/OpenTelemetry/Trace/TraceIdRatioBasedSampler.cs) Custom samplers can be implemented to cover more scenarios: * Samplers should inherit from `OpenTelemetry.Trace.Sampler` (which belongs to the [OpenTelemetry](../../../src/OpenTelemetry/README.md) package), and implement the `ShouldSample` method. * `ShouldSample` should be thread safe, and should not block or take long time, since it will be called on critical code path. ```csharp class MySampler : Sampler { public override SamplingResult ShouldSample(in SamplingParameters samplingParameters) { return new SamplingResult(SamplingDecision.RecordAndSampled); } } ``` A demo sampler is shown [here](./MySampler.cs). ## Resource Detector OpenTelemetry .NET SDK provides a resource detector for detecting resource information from the `OTEL_RESOURCE_ATTRIBUTES` and `OTEL_SERVICE_NAME` environment variables. Custom resource detectors can be implemented: * ResourceDetectors should inherit from `OpenTelemetry.Resources.IResourceDetector`, (which belongs to the [OpenTelemetry](../../../src/OpenTelemetry/README.md) package), and implement the `Detect` method. A demo ResourceDetector is shown [here](./MyResourceDetector.cs). ## References * [Exporter specification](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/sdk.md#span-exporter) * [Processor specification](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/sdk.md#span-processor) * [Resource specification](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/resource/sdk.md) * [Sampler specification](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/sdk.md#sampler)