219 lines
7.5 KiB
Markdown
219 lines
7.5 KiB
Markdown
# Getting Started with Prometheus and Grafana
|
|
|
|
- [Export metrics from the application](#export-metrics-from-the-application)
|
|
- [Check results in the browser](#check-results-in-the-browser)
|
|
- [Collect metrics using Prometheus](#collect-metrics-using-prometheus)
|
|
- [Configuration](#configuration)
|
|
- [Start Prometheus](#start-prometheus)
|
|
- [View results in Prometheus](#view-results-in-prometheus)
|
|
- [Explore metrics using Grafana](#explore-metrics-using-grafana)
|
|
- [Learn more](#learn-more)
|
|
|
|
## Export metrics from the application
|
|
|
|
It is highly recommended to go over the [getting started in 5 minutes - Console
|
|
Application](../getting-started-console/README.md) doc before following along
|
|
this document.
|
|
|
|
Create a new console application and run it:
|
|
|
|
```sh
|
|
dotnet new console --output getting-started-prometheus
|
|
cd getting-started-prometheus
|
|
dotnet run
|
|
```
|
|
|
|
Add a reference to [Prometheus
|
|
Exporter Http Listener](../../../src/OpenTelemetry.Exporter.Prometheus.HttpListener/README.md):
|
|
|
|
```sh
|
|
dotnet add package OpenTelemetry.Exporter.Prometheus.HttpListener --prerelease
|
|
```
|
|
|
|
Now, we are going to make some small tweaks to the example in the
|
|
getting-started metrics `Program.cs` to make the metrics available via
|
|
OpenTelemetry Prometheus Exporter.
|
|
|
|
First, copy and paste everything from getting-started metrics
|
|
[example](../getting-started-console/Program.cs) to the Program.cs file of the
|
|
new console application (getting-started-prometheus) we've created.
|
|
|
|
And replace the below line:
|
|
|
|
```csharp
|
|
.AddConsoleExporter()
|
|
```
|
|
|
|
with
|
|
|
|
```csharp
|
|
.AddPrometheusHttpListener()
|
|
```
|
|
|
|
`PrometheusHttpListener` is a wrapper that contains `PrometheusExporter`. With
|
|
`AddPrometheusHttpListener()`, OpenTelemetry `PrometheusExporter` will export
|
|
data via the endpoint defined by
|
|
[PrometheusHttpListenerOptions.UriPrefixes](../../../src/OpenTelemetry.Exporter.Prometheus.HttpListener/README.md#uriprefixes),
|
|
which is `http://localhost:9464/` by default.
|
|
|
|
```mermaid
|
|
graph LR
|
|
|
|
subgraph SDK
|
|
MeterProvider
|
|
MetricReader[BaseExportingMetricReader]
|
|
PrometheusHttpListener["PrometheusHttpListener<br/>(http://localhost:9464/)"]
|
|
end
|
|
|
|
subgraph API
|
|
Instrument["Meter(#quot;MyCompany.MyProduct.MyLibrary#quot;, #quot;1.0#quot;)<br/>Counter(#quot;MyFruitCounter#quot;)"]
|
|
end
|
|
|
|
Instrument --> | Measurements | MeterProvider
|
|
|
|
MeterProvider --> | Metrics | MetricReader --> | Pull | PrometheusHttpListener
|
|
```
|
|
|
|
Also, for our learning purpose, use a while-loop to keep increasing the counter
|
|
value until any key is pressed.
|
|
|
|
```csharp
|
|
Console.WriteLine("Press any key to exit");
|
|
while (!Console.KeyAvailable)
|
|
{
|
|
Thread.Sleep(1000);
|
|
MyFruitCounter.Add(1, new("name", "apple"), new("color", "red"));
|
|
MyFruitCounter.Add(2, new("name", "lemon"), new("color", "yellow"));
|
|
MyFruitCounter.Add(1, new("name", "lemon"), new("color", "yellow"));
|
|
...
|
|
...
|
|
...
|
|
}
|
|
```
|
|
|
|
After the above modifications, now our `Program.cs` should look like [this](./Program.cs).
|
|
|
|
### Check results in the browser
|
|
|
|
Start the application and keep it running. Now we should be able to see the
|
|
metrics at [http://localhost:9464/metrics](http://localhost:9464/metrics) from a
|
|
web browser:
|
|
|
|
|
|
|
|
Now, we understand how we can configure `PrometheusHttpListener` to export metrics.
|
|
Next, we are going to learn about how to use Prometheus to collect the metrics.
|
|
|
|
## Collect metrics using Prometheus
|
|
|
|
Follow the [first steps](https://prometheus.io/docs/introduction/first_steps/)
|
|
to download the [latest release](https://prometheus.io/download/) of Prometheus.
|
|
|
|
### Configuration
|
|
|
|
After finished downloading, extract it to a local location that's easy to
|
|
access. We will find the default Prometheus configuration YAML file in the
|
|
folder, named `prometheus.yml`.
|
|
|
|
Let's create a new file in the same location as where `prometheus.yml` locates,
|
|
and named the new file as `otel.yml` for this exercise. Then, copy and paste the
|
|
entire content below into the `otel.yml` file we have created just now.
|
|
|
|
```yaml
|
|
global:
|
|
scrape_interval: 10s
|
|
evaluation_interval: 10s
|
|
scrape_configs:
|
|
- job_name: "otel"
|
|
static_configs:
|
|
- targets: ["localhost:9464"]
|
|
```
|
|
|
|
### Start Prometheus
|
|
|
|
Follow the instructions from
|
|
[starting-prometheus](https://prometheus.io/docs/introduction/first_steps/#starting-prometheus)
|
|
to start the Prometheus server and verify it has been started successfully.
|
|
|
|
Please note that we will need pass in `otel.yml` file as the argument:
|
|
|
|
```console
|
|
./prometheus --config.file=otel.yml
|
|
```
|
|
|
|
### View results in Prometheus
|
|
|
|
To use the graphical interface for viewing our metrics with Prometheus, navigate
|
|
to [http://localhost:9090/graph](http://localhost:9090/graph), and type
|
|
`MyFruitCounter` in the expression bar of the UI; finally, click the execute
|
|
button.
|
|
|
|
We should be able to see the following chart from the browser:
|
|
|
|
|
|
|
|
From the legend, we can see that the `instance` name and the `job` name are the
|
|
values we have set in `otel.yml`.
|
|
|
|
Congratulations!
|
|
|
|
Now we know how to configure Prometheus server and deploy OpenTelemetry
|
|
`PrometheusHttpListener` to export our metrics. Next, we are going to explore a tool
|
|
called Grafana, which has powerful visualizations for the metrics.
|
|
|
|
## Explore metrics using Grafana
|
|
|
|
[Install Grafana](https://grafana.com/docs/grafana/latest/installation/).
|
|
|
|
Start the standalone Grafana server (`grafana-server.exe` or
|
|
`./bin/grafana-server`, depending on the operating system). Then, use the
|
|
browser to navigate to [http://localhost:3000/](http://localhost:3000/).
|
|
|
|
Follow the instructions in the Grafana getting started
|
|
[doc](https://grafana.com/docs/grafana/latest/getting-started/getting-started/#step-2-log-in)
|
|
to log in.
|
|
|
|
After successfully logging in, hover on the Configuration icon
|
|
on the panel at the left hand side, and click on Plugins.
|
|
Find and click on the Prometheus plugin. Next click on
|
|
`Create a Prometheus data source` button. Type in the default endpoint of
|
|
Prometheus as suggested by the UI as the value for the URI.
|
|
|
|
```console
|
|
http://localhost:9090
|
|
```
|
|
|
|
At the bottom of the page click `Save & test` to ensure the data source is
|
|
working. Then, click on the `Explore` button - we should be able to write
|
|
some queries to explore our metrics now!
|
|
|
|
Feel free to find some handy PromQL
|
|
[here](https://promlabs.com/promql-cheat-sheet/).
|
|
|
|
In the below example, the query targets to find out what is the per-second rate
|
|
of increase of myFruitCounter over the past 5 minutes:
|
|
|
|
|
|
|
|
```mermaid
|
|
graph TD
|
|
|
|
subgraph Prometheus
|
|
PrometheusScraper
|
|
PrometheusDatabase
|
|
end
|
|
|
|
PrometheusHttpListener["PrometheusHttpListener<br/>(listening at #quot;http://localhost:9464/#quot;)"] -->|HTTP GET| PrometheusScraper{{"Prometheus scraper<br/>(polling #quot;http://localhost:9464/metrics#quot; every 10 seconds)"}}
|
|
PrometheusScraper --> PrometheusDatabase[("Prometheus TSDB (time series database)")]
|
|
PrometheusDatabase -->|http://localhost:9090/graph| PrometheusUI["Browser<br/>(Prometheus Dashboard)"]
|
|
PrometheusDatabase -->|http://localhost:9090/api/| Grafana[Grafana Server]
|
|
Grafana -->|http://localhost:3000/dashboard| GrafanaUI["Browser<br/>(Grafana Dashboard)"]
|
|
```
|
|
|
|
## Learn more
|
|
|
|
- [What is Prometheus?](https://prometheus.io/docs/introduction/overview/)
|
|
- [Grafana support for
|
|
Prometheus](https://prometheus.io/docs/visualization/grafana/#creating-a-prometheus-graph)
|