huggingface / huggingface/dataset-viewer
Fix gauge metrics in api and admin services
- Dominant language
- Python
- Stars
- 899
- Forks
- 130
- Avg merge
- 9h 26m
- Merged PRs (30d)
- 14
Description
The API and admin services run using uvicorn with multiple workers. We set the `PROMETHEUS_MULTIPROC_DIR` environment variable (to `/tmp`), as described in https://github.com/prometheus/client_python#multiprocess-mode-eg-gunicorn, which allows every worker to write in a dedicated file. See, for example, the content of `/tmp` for one of the pods of the admin service:

Each of the nine workers has its file for every metric type: `counter`, `gauge_all`, `gauge_liveall`, `histogram`.
But, as described in https://github.com/prometheus/client_python#multiprocess-mode-eg-gunicorn, the "live" gauge metrics are special: they should only count the live workers, and to do that, the files starting with `gauge_live` [should be deleted](https://github.com/prometheus/client_python/blob/30f83196ac1f1a1a2626da9d724d83955aff79b1/prometheus_client/multiprocess.py#L157) on a worker child exit. But we don't do it, since uvicorn does not allow to call a specific function on child exit.
Two solutions:
- [run the services with gunicorn](https://www.uvicorn.org/deployment/)
- or, avoid using "live" gauges.
Contributor guide
Assessment
This issue has not been assessed yet.