Skip to content

Commit

Permalink
engine: update prometheus metrics collection
Browse files Browse the repository at this point in the history
Signed-off-by: David Karlsson <[email protected]>
  • Loading branch information
dvdksn committed Nov 24, 2023
1 parent e6eec3f commit cb0c763
Show file tree
Hide file tree
Showing 7 changed files with 65 additions and 73 deletions.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file removed content/config/daemon/images/prometheus-targets.png
Binary file not shown.
Binary file not shown.
138 changes: 65 additions & 73 deletions content/config/daemon/prometheus.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,9 +9,7 @@ aliases:
---

[Prometheus](https://prometheus.io/) is an open-source systems monitoring and
alerting toolkit. You can configure Docker as a Prometheus target. This topic
shows you how to configure Docker, set up Prometheus to run as a Docker
container, and monitor your Docker instance using Prometheus.
alerting toolkit. You can configure Docker as a Prometheus target.

> **Warning**
>
Expand All @@ -22,44 +20,38 @@ container, and monitor your Docker instance using Prometheus.
Currently, you can only monitor Docker itself. You can't currently monitor your
application using the Docker target.

## Prerequisites
## Example

1. One or more Docker engines are joined into a Docker Swarm, using `docker
swarm init` on one manager and `docker swarm join` on other managers and
worker nodes.
2. You need an internet connection to pull the Prometheus image.
The following example shows you how to configure your Docker daemon, set up
Prometheus to run as a container on your local machine, and monitor your Docker
instance using Prometheus.

## Configure Docker
### Configure the daemon

To configure the Docker daemon as a Prometheus target, you need to specify the
`metrics-address`. The best way to do this is via the `daemon.json`, which is
located at one of the following locations by default. If the file doesn't
exist, create it.
`metrics-address` in the `daemon.json` configuration file. This daemon expects
the file to be located at one of the following locations by default. If the
file doesn't exist, create it.

- **Linux**: `/etc/docker/daemon.json`
- **Windows Server**: `C:\ProgramData\docker\config\daemon.json`
- **Docker Desktop**: Open the Docker Desktop settings and select **Docker Engine**.
- **Docker Desktop**: Open the Docker Desktop settings and select **Docker Engine** to edit the file.

If the file is currently empty, paste the following:
Add the following configuration:

```json
{
"metrics-addr": "127.0.0.1:9323"
}
```

If the file isn't empty, add the new key, making sure that the resulting
file is valid JSON. Be careful that every line ends with a comma (`,`) except
for the last line.
Save the file, or in the case of Docker Desktop for Mac or Docker Desktop for
Windows, save the configuration. Restart Docker.

Save the file, or in the case of Docker Desktop for Mac or Docker Desktop for Windows, save the
configuration. Restart Docker.
Docker now exposes Prometheus-compatible metrics on port 9323 on the loopback
interface.

Docker now exposes Prometheus-compatible metrics on port 9323.

## Configure and run Prometheus

Prometheus runs as a Docker service on a Swarm.
### Create a Prometheus configuration

Copy the following configuration file and save it to a location of your choice,
for example `/tmp/prometheus.yml`. This is a stock Prometheus configuration file,
Expand All @@ -86,90 +78,90 @@ rule_files:
# Here it's Prometheus itself.
scrape_configs:
# The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
- job_name: "prometheus"
- job_name: prometheus

# metrics_path defaults to '/metrics'
# scheme defaults to 'http'.

static_configs:
- targets: ["host.docker.internal:9090"]
- targets: ["localhost:9090"]

- job_name:
"docker"
- job_name: docker
# metrics_path defaults to '/metrics'
# scheme defaults to 'http'.

static_configs:
- targets: ["localhost:9323"]
- targets: ["host.docker.internal:9323"]
```
Next, start a single-replica Prometheus service using this configuration.
### Run Prometheus in a container
- If you're using Docker Desktop, run:
Next, start a Prometheus container using this configuration.
```console
$ docker service create --replicas 1 --name my-prometheus \
--mount type=bind,source=/tmp/prometheus.yml,destination=/etc/prometheus/prometheus.yml \
--publish published=9090,target=9090,protocol=tcp \
prom/prometheus
```
```console
$ docker run --name my-prometheus \
--mount type=bind,source=/tmp/prometheus.yml,destination=/etc/prometheus/prometheus.yml \
-p 9090:9090 \
--add-host host.docker.internal:host-gateway \
prom/prometheus
```

- If you're using Docker Engine without Docker Desktop, run:
If you're using Docker Desktop, the `--add-host` flag is optional. This flag
makes sure that the host's internal IP gets exposed to the Prometheus
container. Docker Desktop does this by default. The host IP is exposed as the
`host.docker.internal` hostname. This matches the configuration defined in
`prometheus.yml` in the previous step.

```console
$ docker service create --replicas 1 --name my-prometheus \
--mount type=bind,source=/tmp/prometheus.yml,destination=/etc/prometheus/prometheus.yml \
--publish published=9090,target=9090,protocol=tcp \
--add-host host.docker.internal:host-gateway \
prom/prometheus
```
### Open the Prometheus Dashboard

Verify that the Docker target is listed at `http://localhost:9090/targets/`.

![Prometheus targets page](images/prometheus-targets.png)
![Prometheus targets page](images/prometheus-targets.webp)

You can't access the endpoint URLs directly if you use Docker Desktop
for Mac or Docker Desktop for Windows.
> **Note**
>
> You can't access the endpoint URLs on this page directly if you use Docker
> Desktop.
## Use Prometheus
### Use Prometheus

Create a graph. Select the **Graphs** link in the Prometheus UI. Choose a metric
from the combo box to the right of the **Execute** button, and click
**Execute**. The screenshot below shows the graph for
`engine_daemon_network_actions_seconds_count`.

![Prometheus engine_daemon_network_actions_seconds_count report](images/prometheus-graph_idle.png)
![Idle Prometheus report](images/prometheus-graph_idle.webp)

The above graph shows a pretty idle Docker instance. Your graph might look
different if you are running active workloads.
The graph shows a pretty idle Docker instance, unless you're already running
active workloads on your system.

To make the graph more interesting, create some network actions by starting
a service with 10 tasks that just ping Docker non-stop (you can change the
ping target to anything you like):
To make the graph more interesting, run a container that uses some network
actions by starting downloading some packages using a package manager:

```console
$ docker service create \
--replicas 10 \
--name ping_service \
alpine ping docker.com
$ docker run --rm alpine apk add git make musl-dev go
```

Wait a few minutes (the default scrape interval is 15 seconds) and reload
your graph.
Wait a few seconds (the default scrape interval is 15 seconds) and reload your
graph. You should see an uptick in the graph, showing the increased network
traffic caused by the container you just ran.

![Prometheus engine_daemon_network_actions_seconds_count report](images/prometheus-graph_load.png)
![Prometheus report showing traffic](images/prometheus-graph_load.webp)

When you are ready, stop and remove the `ping_service` service, so that you
aren't flooding a host with pings for no reason.
## Next steps

```console
$ docker service remove ping_service
The example provided here shows how to run Prometheus as a container on your
local system. In practice, you'll probably be running Prometheus on another
system or as a cloud service somewhere. You can set up the Docker daemon as a
Prometheus target in such contexts too. Configure the `metrics-addr` of the
daemon and add the address of the daemon as a scrape endpoint in your
Prometheus configuration.

```yaml
- job_name: docker
static_configs:
- targets: ["docker.daemon.example:<PORT>"]
```
Wait a few minutes and you should see that the graph falls back to the idle
level.

## Next steps

- Read the [Prometheus documentation](https://prometheus.io/docs/introduction/overview/)
- Set up some [alerts](https://prometheus.io/docs/alerting/overview/)
For more information about Prometheus, refer to the
[Prometheus documentation](https://prometheus.io/docs/introduction/overview/)

0 comments on commit cb0c763

Please sign in to comment.