opentelemetry-dotnet/docs/metrics/getting-started-prometheus-grafana

Getting Started with Prometheus and Grafana

• Export metrics from the application
  • Check results in the browser
• Collect metrics using Prometheus
  • Configuration
  • Start Prometheus
  • View results in Prometheus
• Explore metrics using Grafana
• Learn more

Export metrics from the application

It is highly recommended to go over the getting-started doc before following along this document.

Create a new console application and run it:

dotnet new console --output getting-started-prometheus
cd getting-started-prometheus
dotnet run

Add a reference to Prometheus Exporter Http Listener:

dotnet add package --prerelease OpenTelemetry.Exporter.Prometheus.HttpListener

Now, we are going to make some small tweaks to the example in the getting-started metrics Program.cs to make the metrics available via OpenTelemetry Prometheus Exporter.

First, copy and paste everything from getting-started metrics example to the Program.cs file of the new console application (getting-started-prometheus) we've created.

And replace the below line:

.AddConsoleExporter()

with

.AddPrometheusHttpListener()

PrometheusHttpListener is a wrapper that contains PrometheusExporter. With AddPrometheusHttpListener(), OpenTelemetry PrometheusExporter will export data via the endpoint defined by PrometheusHttpListenerOptions.UriPrefixes, which is http://localhost:9464/ by default.

graph LR

subgraph SDK
  MeterProvider
  MetricReader[BaseExportingMetricReader]
  PrometheusHttpListener["PrometheusHttpListener<br/>(http://localhost:9464/)"]
end

subgraph API
  Instrument["Meter(#quot;MyCompany.MyProduct.MyLibrary#quot;, #quot;1.0#quot;)<br/>Counter(#quot;MyFruitCounter#quot;)"]
end

Instrument --> | Measurements | MeterProvider

MeterProvider --> | Metrics | MetricReader --> | Pull | PrometheusHttpListener

Also, for our learning purpose, use a while-loop to keep increasing the counter value until any key is pressed.

Console.WriteLine("Press any key to exit");
while (!Console.KeyAvailable)
{
    Thread.Sleep(1000);
    MyFruitCounter.Add(1, new("name", "apple"), new("color", "red"));
    MyFruitCounter.Add(2, new("name", "lemon"), new("color", "yellow"));
    MyFruitCounter.Add(1, new("name", "lemon"), new("color", "yellow"));
    ...
    ...
    ...
}

After the above modifications, now our Program.cs should look like this.

Check results in the browser

Start the application and keep it running. Now we should be able to see the metrics at http://localhost:9464/metrics from a web browser:

Now, we understand how we can configure PrometheusHttpListener to export metrics. Next, we are going to learn about how to use Prometheus to collect the metrics.

Collect metrics using Prometheus

Follow the first steps to download the latest release of Prometheus.

Configuration

After finished downloading, extract it to a local location that's easy to access. We will find the default Prometheus configuration YAML file in the folder, named prometheus.yml.

Let's create a new file in the same location as where prometheus.yml locates, and named the new file as otel.yml for this exercise. Then, copy and paste the entire content below into the otel.yml file we have created just now.

global:
  scrape_interval: 10s
  evaluation_interval: 10s
scrape_configs:
  - job_name: "otel"
    static_configs:
      - targets: ["localhost:9464"]

Start Prometheus

Follow the instructions from starting-prometheus to start the Prometheus server and verify it has been started successfully.

Please note that we will need pass in otel.yml file as the argument:

./prometheus --config.file=otel.yml

View results in Prometheus

To use the graphical interface for viewing our metrics with Prometheus, navigate to http://localhost:9090/graph, and type MyFruitCounter in the expression bar of the UI; finally, click the execute button.

We should be able to see the following chart from the browser:

From the legend, we can see that the instance name and the job name are the values we have set in otel.yml.

Congratulations!

Now we know how to configure Prometheus server and deploy OpenTelemetry PrometheusHttpListener to export our metrics. Next, we are going to explore a tool called Grafana, which has powerful visualizations for the metrics.

Explore metrics using Grafana

Install Grafana.

Start the standalone Grafana server (grafana-server.exe or ./bin/grafana-server, depending on the operating system). Then, use the browser to navigate to http://localhost:3000/.

Follow the instructions in the Grafana getting started doc to log in.

After successfully logging in, click on the Configuration icon on the panel at the left hand side, and click on Prometheus. Type in the default endpoint of Prometheus as suggested by the UI as the value for the URI.

http://localhost:9090

Then, click on the Explore icon on the left panel of the website - we should be able to write some queries to explore our metrics now!

Feel free to find some handy PromQL here.

In the below example, the query targets to find out what is the per-second rate of increase of myFruitCounter over the past 5 minutes:

graph TD

subgraph Prometheus
  PrometheusScraper
  PrometheusDatabase
end

PrometheusHttpListener["PrometheusHttpListener<br/>(listening at #quot;http://localhost:9464/#quot;)"] -->|HTTP GET| PrometheusScraper{{"Prometheus scraper<br/>(polling #quot;http://localhost:9464/metrics#quot; every 10 seconds)"}}
PrometheusScraper --> PrometheusDatabase[("Prometheus TSDB (time series database)")]
PrometheusDatabase -->|http://localhost:9090/graph| PrometheusUI["Browser<br/>(Prometheus Dashboard)"]
PrometheusDatabase -->|http://localhost:9090/api/| Grafana[Grafana Server]
Grafana -->|http://localhost:3000/dashboard| GrafanaUI["Browser<br/>(Grafana Dashboard)"]

Learn more

• What is Prometheus?
• Grafana support for Prometheus