engine: update prometheus metrics collection

Signed-off-by: David Karlsson <35727626+dvdksn@users.noreply.github.com>
This commit is contained in:
David Karlsson 2023-11-24 12:04:13 +01:00
parent e6eec3f612
commit cb0c763b66
7 changed files with 66 additions and 74 deletions

Binary file not shown.

Before

Width:  |  Height:  |  Size: 171 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 73 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 173 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 74 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 131 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 65 KiB

View File

@ -9,9 +9,7 @@ aliases:
--- ---
[Prometheus](https://prometheus.io/) is an open-source systems monitoring and [Prometheus](https://prometheus.io/) is an open-source systems monitoring and
alerting toolkit. You can configure Docker as a Prometheus target. This topic alerting toolkit. You can configure Docker as a Prometheus target.
shows you how to configure Docker, set up Prometheus to run as a Docker
container, and monitor your Docker instance using Prometheus.
> **Warning** > **Warning**
> >
@ -22,25 +20,24 @@ container, and monitor your Docker instance using Prometheus.
Currently, you can only monitor Docker itself. You can't currently monitor your Currently, you can only monitor Docker itself. You can't currently monitor your
application using the Docker target. application using the Docker target.
## Prerequisites ## Example
1. One or more Docker engines are joined into a Docker Swarm, using `docker The following example shows you how to configure your Docker daemon, set up
swarm init` on one manager and `docker swarm join` on other managers and Prometheus to run as a container on your local machine, and monitor your Docker
worker nodes. instance using Prometheus.
2. You need an internet connection to pull the Prometheus image.
## Configure Docker ### Configure the daemon
To configure the Docker daemon as a Prometheus target, you need to specify the To configure the Docker daemon as a Prometheus target, you need to specify the
`metrics-address`. The best way to do this is via the `daemon.json`, which is `metrics-address` in the `daemon.json` configuration file. This daemon expects
located at one of the following locations by default. If the file doesn't the file to be located at one of the following locations by default. If the
exist, create it. file doesn't exist, create it.
- **Linux**: `/etc/docker/daemon.json` - **Linux**: `/etc/docker/daemon.json`
- **Windows Server**: `C:\ProgramData\docker\config\daemon.json` - **Windows Server**: `C:\ProgramData\docker\config\daemon.json`
- **Docker Desktop**: Open the Docker Desktop settings and select **Docker Engine**. - **Docker Desktop**: Open the Docker Desktop settings and select **Docker Engine** to edit the file.
If the file is currently empty, paste the following: Add the following configuration:
```json ```json
{ {
@ -48,18 +45,13 @@ If the file is currently empty, paste the following:
} }
``` ```
If the file isn't empty, add the new key, making sure that the resulting Save the file, or in the case of Docker Desktop for Mac or Docker Desktop for
file is valid JSON. Be careful that every line ends with a comma (`,`) except Windows, save the configuration. Restart Docker.
for the last line.
Save the file, or in the case of Docker Desktop for Mac or Docker Desktop for Windows, save the Docker now exposes Prometheus-compatible metrics on port 9323 on the loopback
configuration. Restart Docker. interface.
Docker now exposes Prometheus-compatible metrics on port 9323. ### Create a Prometheus configuration
## Configure and run Prometheus
Prometheus runs as a Docker service on a Swarm.
Copy the following configuration file and save it to a location of your choice, Copy the following configuration file and save it to a location of your choice,
for example `/tmp/prometheus.yml`. This is a stock Prometheus configuration file, for example `/tmp/prometheus.yml`. This is a stock Prometheus configuration file,
@ -86,90 +78,90 @@ rule_files:
# Here it's Prometheus itself. # Here it's Prometheus itself.
scrape_configs: scrape_configs:
# The job name is added as a label `job=<job_name>` to any timeseries scraped from this config. # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
- job_name: "prometheus" - job_name: prometheus
# metrics_path defaults to '/metrics' # metrics_path defaults to '/metrics'
# scheme defaults to 'http'. # scheme defaults to 'http'.
static_configs: static_configs:
- targets: ["host.docker.internal:9090"] - targets: ["localhost:9090"]
- job_name: - job_name: docker
"docker"
# metrics_path defaults to '/metrics' # metrics_path defaults to '/metrics'
# scheme defaults to 'http'. # scheme defaults to 'http'.
static_configs: static_configs:
- targets: ["localhost:9323"] - targets: ["host.docker.internal:9323"]
``` ```
Next, start a single-replica Prometheus service using this configuration. ### Run Prometheus in a container
- If you're using Docker Desktop, run: Next, start a Prometheus container using this configuration.
```console ```console
$ docker service create --replicas 1 --name my-prometheus \ $ docker run --name my-prometheus \
--mount type=bind,source=/tmp/prometheus.yml,destination=/etc/prometheus/prometheus.yml \ --mount type=bind,source=/tmp/prometheus.yml,destination=/etc/prometheus/prometheus.yml \
--publish published=9090,target=9090,protocol=tcp \ -p 9090:9090 \
prom/prometheus --add-host host.docker.internal:host-gateway \
``` prom/prometheus
```
- If you're using Docker Engine without Docker Desktop, run: If you're using Docker Desktop, the `--add-host` flag is optional. This flag
makes sure that the host's internal IP gets exposed to the Prometheus
container. Docker Desktop does this by default. The host IP is exposed as the
`host.docker.internal` hostname. This matches the configuration defined in
`prometheus.yml` in the previous step.
```console ### Open the Prometheus Dashboard
$ docker service create --replicas 1 --name my-prometheus \
--mount type=bind,source=/tmp/prometheus.yml,destination=/etc/prometheus/prometheus.yml \
--publish published=9090,target=9090,protocol=tcp \
--add-host host.docker.internal:host-gateway \
prom/prometheus
```
Verify that the Docker target is listed at `http://localhost:9090/targets/`. Verify that the Docker target is listed at `http://localhost:9090/targets/`.
![Prometheus targets page](images/prometheus-targets.png) ![Prometheus targets page](images/prometheus-targets.webp)
You can't access the endpoint URLs directly if you use Docker Desktop > **Note**
for Mac or Docker Desktop for Windows. >
> You can't access the endpoint URLs on this page directly if you use Docker
> Desktop.
## Use Prometheus ### Use Prometheus
Create a graph. Select the **Graphs** link in the Prometheus UI. Choose a metric Create a graph. Select the **Graphs** link in the Prometheus UI. Choose a metric
from the combo box to the right of the **Execute** button, and click from the combo box to the right of the **Execute** button, and click
**Execute**. The screenshot below shows the graph for **Execute**. The screenshot below shows the graph for
`engine_daemon_network_actions_seconds_count`. `engine_daemon_network_actions_seconds_count`.
![Prometheus engine_daemon_network_actions_seconds_count report](images/prometheus-graph_idle.png) ![Idle Prometheus report](images/prometheus-graph_idle.webp)
The above graph shows a pretty idle Docker instance. Your graph might look The graph shows a pretty idle Docker instance, unless you're already running
different if you are running active workloads. active workloads on your system.
To make the graph more interesting, create some network actions by starting To make the graph more interesting, run a container that uses some network
a service with 10 tasks that just ping Docker non-stop (you can change the actions by starting downloading some packages using a package manager:
ping target to anything you like):
```console ```console
$ docker service create \ $ docker run --rm alpine apk add git make musl-dev go
--replicas 10 \
--name ping_service \
alpine ping docker.com
``` ```
Wait a few minutes (the default scrape interval is 15 seconds) and reload Wait a few seconds (the default scrape interval is 15 seconds) and reload your
your graph. graph. You should see an uptick in the graph, showing the increased network
traffic caused by the container you just ran.
![Prometheus engine_daemon_network_actions_seconds_count report](images/prometheus-graph_load.png) ![Prometheus report showing traffic](images/prometheus-graph_load.webp)
When you are ready, stop and remove the `ping_service` service, so that you
aren't flooding a host with pings for no reason.
```console
$ docker service remove ping_service
```
Wait a few minutes and you should see that the graph falls back to the idle
level.
## Next steps ## Next steps
- Read the [Prometheus documentation](https://prometheus.io/docs/introduction/overview/) The example provided here shows how to run Prometheus as a container on your
- Set up some [alerts](https://prometheus.io/docs/alerting/overview/) local system. In practice, you'll probably be running Prometheus on another
system or as a cloud service somewhere. You can set up the Docker daemon as a
Prometheus target in such contexts too. Configure the `metrics-addr` of the
daemon and add the address of the daemon as a scrape endpoint in your
Prometheus configuration.
```yaml
- job_name: docker
static_configs:
- targets: ["docker.daemon.example:<PORT>"]
```
For more information about Prometheus, refer to the
[Prometheus documentation](https://prometheus.io/docs/introduction/overview/)