dask / dask/distributed

`/metrics` sometimes miss dask specific metrics

Open
#9,096 2 comments 0 reactions 0 assignees View on GitHub
bug help wanted
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

**Describe the issue**:

Collecting metrics with Prometeus sometimes misses `dask_*` specific mertrics. It happens because `/metrics` endpoints does not expose them and return only default Prometheus metrics:
```
# HELP python_gc_objects_collected_total Objects collected during gc
# TYPE python_gc_objects_collected_total counter
python_gc_objects_collected_total{generation="0"} 24345.0
python_gc_objects_collected_total{generation="1"} 12966.0
python_gc_objects_collected_total{generation="2"} 4013.0
# HELP python_gc_objects_uncollectable_total Uncollectable objects found during GC
# TYPE python_gc_objects_uncollectable_total counter
python_gc_objects_uncollectable_total{generation="0"} 0.0
python_gc_objects_uncollectable_total{generation="1"} 0.0
python_gc_objects_uncollectable_total{generation="2"} 0.0
# HELP python_gc_collections_total Number of times this generation was collected
# TYPE python_gc_collections_total counter
python_gc_collections_total{generation="0"} 78769.0
python_gc_collections_total{generation="1"} 7160.0
python_gc_collections_total{generation="2"} 197.0
# HELP python_info Python platform information
# TYPE python_info gauge
python_info{implementation="CPython",major="3",minor="11",patchlevel="11",version="3.11.11"} 1.0
# HELP process_virtual_memory_bytes Virtual memory size in bytes.
# TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 6.136066048e+09
# HELP process_resident_memory_bytes Resident memory size in bytes.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 2.284679168e+09
# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1.75200250106e+09
# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 1183.52
# HELP process_open_fds Number of open file descriptors.
# TYPE process_open_fds gauge
process_open_fds 126.0
# HELP process_max_fds Maximum number of open file descriptors.
# TYPE process_max_fds gauge
process_max_fds 1.048576e+06
```

**Minimal Complete Verifiable Example**:

Steps to reproduce
1. Create a simple script:
```bash
#!/bin/bash
#!/bin/bash
WORKER_IP="$1"
while true; do
curl http://${WORKER_IP}:8787/metrics
echo "------------------------------------------"
done
```
3. Start new cluster in the cloud. It is important to have a time before the worker created and started to listed on port `8787`. It is important to force worker to listed on port `8787` by providing the following worker options:
```python
from dask_cloudprovider.gcp import GCPCluster

GCPCluster(
# ...
"worker_options": {
# Required to expose Prometheus metrics. See https://github.com/dask/distributed/issues/4160
"dashboard_address": "0.0.0.0:8787",
},
# ...
)
```
4. During the workers being starting and before they are ready, you have to pick one worker IP and pass it into script (e.g., 1.2.3.4). Run the script before worker is ready to receive requests:
```bash
sh curl_metrics.sh 1.2.3.4
```
You should see that curl can not connect to the port:
```
curl: (7) Failed to connect to 1.2.3.4 port 8787 after 0 ms: Connection refused

curl: (7) Failed to connect to 1.2.3.4 port 8787 after 0 ms: Connection refused

curl: (7) Failed to connect to 1.2.3.4 port 8787 after 0 ms: Connection refused

...
```
4. Right after the worker started you'll receive the list of metrics that miss `dask_worker_*` metrics:
```
...
# HELP process_max_fds Maximum number of open file descriptors.
# TYPE process_max_fds gauge
process_max_fds 1.048576e+06
------------------------------------------
# HELP python_gc_objects_collected_total Objects collected during gc
# TYPE python_gc_objects_collected_total counter
python_gc_objects_collected_total{generation="0"} 216513.0
python_gc_objects_collected_total{generation="1"} 29043.0
python_gc_objects_collected_total{generation="2"} 2142.0
# HELP python_gc_objects_uncollectable_total Uncollectable objects found during GC
# TYPE python_gc_objects_uncollectable_total counter
python_gc_objects_uncollectable_total{generation="0"} 0.0
python_gc_objects_uncollectable_total{generation="1"} 0.0
python_gc_objects_uncollectable_total{generation="2"} 0.0
# HELP python_gc_collections_total Number of times this generation was collected
# TYPE python_gc_collections_total counter
python_gc_collections_total{generation="0"} 1535.0
python_gc_collections_total{generation="1"} 139.0
python_gc_collections_total{generation="2"} 11.0
# HELP python_info Python platform information
# TYPE python_info gauge
python_info{implementation="CPython",major="3",minor="11",patchlevel="11",version="3.11.11"} 1.0
# HELP process_virtual_memory_bytes Virtual memory size in bytes.
# TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 1.872535552e+09
# HELP process_resident_memory_bytes Resident memory size in bytes.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 3.70044928e+08
# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1.75204228461e+09
# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 116.95
# HELP process_open_fds Number of open file descriptors.
# TYPE process_open_fds gauge
process_open_fds 33.0
# HELP process_max_fds Maximum number of open file descriptors.
# TYPE process_max_fds gauge
process_max_fds 1.048576e+06
------------------------------------------
```
5. Check other workers. You will see that other workers return a complete list of metrics.

**Environment**:

- Dask version: 2024.12.1
- Python version: 3.11
- Operating System: WSL
- Install method (conda, pip, source): poetry

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.