huggingface / huggingface/dataset-viewer

Fix gauge metrics in api and admin services

Open
#889 10 comments 0 reactions 0 assignees View on GitHub
bug metrics P2
Dominant language
Python
Stars
899
Forks
130
Avg merge
9h 26m
Merged PRs (30d)
14

Description

The API and admin services run using uvicorn with multiple workers. We set the `PROMETHEUS_MULTIPROC_DIR` environment variable (to `/tmp`), as described in https://github.com/prometheus/client_python#multiprocess-mode-eg-gunicorn, which allows every worker to write in a dedicated file. See, for example, the content of `/tmp` for one of the pods of the admin service:

Capture d’écran 2023-03-03 à 11 53 32

Each of the nine workers has its file for every metric type: `counter`, `gauge_all`, `gauge_liveall`, `histogram`.

But, as described in https://github.com/prometheus/client_python#multiprocess-mode-eg-gunicorn, the "live" gauge metrics are special: they should only count the live workers, and to do that, the files starting with `gauge_live` [should be deleted](https://github.com/prometheus/client_python/blob/30f83196ac1f1a1a2626da9d724d83955aff79b1/prometheus_client/multiprocess.py#L157) on a worker child exit. But we don't do it, since uvicorn does not allow to call a specific function on child exit.

Two solutions:
- [run the services with gunicorn](https://www.uvicorn.org/deployment/)
- or, avoid using "live" gauges.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.