prometheus / prometheus/client_python
Metric without `labelnames` causes issues in when MULTIPROC is enabled
还没有人认领这个 Issue。
- 主要语言
- Python
- 星标
- 4.4k
- 派生
- 876
- 平均合并
- 8 天 4 小时
- 30 天内合并 PR
- 1
描述
Hello,
I noticed (see https://github.com/prometheus/client_python/issues/902#issuecomment-3013566209 for example ) that when
- having a metric defined without a label
- multiprocess mode is enabled (i.e.
PROMETHEUS_MULTIPROC_DIRis set ) - an app has a single process
then this causes issues, and some metrics will not be reported.
It's not exactly clear to me what is happening, but an indication that something is wrong is that a prometheus db file is created at load time (i.e. when metrics are defined, before they are set).
Here's what I have in more details:
Observations in a prod application:
- Some histogram metrics set in a threaded celery worker where
PROMETHEUS_MULTIPROC_DIRare not reported - No such issue in pre-fork workers
- The issue was resolved by adding the
labelnamesargument to a metric where it was missing (which was an histogram as well)
Other observations:
- When starting the app, a prom db file is created for the metric that was missing labelnames (before any measurement is made)
Hypothesis:
- in a celery worker in pre-fork mode, the process creating the first db file is not the same process were metrics are set afterwards, so there is no "collision" (because of the pid suffix)
- in a celery worker in threaded mode, there is a single process creating the first db file and setting the metrics, and somehow collisions happens and some metrics are not reported
Values set at startup:
$ curl localhost:8000
# HELP test_histogram_no_label test Histogram
# TYPE test_histogram_no_label histogram
test_histogram_no_label_sum 0.0
test_histogram_no_label_bucket{le="1.0"} 0.0
test_histogram_no_label_bucket{le="2.0"} 0.0
test_histogram_no_label_bucket{le="+Inf"} 0.0
test_histogram_no_label_count 0.0
It seems to me that a simple way to address this would be to raise when a metric is defined without labelnames, as labelnames are mandatory anyway.
贡献指南
从这里开始
- 先读完整个 Issue,再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库,在一个分支上完成修改。
- 提交 Pull Request,并在描述里引用这个 Issue 编号。
调研方向
首先,在启用 PROMETHEUS_MULTIPROC_DIR 的情况下复现单进程、多线程设置,并比较使用和不使用 labelnames 定义的指标。观察指标初始化、创建的数据库文件和 exposition 输出,然后跟踪所涉及的 multiprocess 处理。完成的标准是理解其行为、指标能够一致地报告,并明确决定是否要验证缺失的 labelnames。
由索引模型根据 Issue 内容生成。
评估
- 技术栈
- python
- 领域
- observability-sre
- Issue 类型
- 缺陷
- 难度
- 4/5
- 预计耗时
- 3-5 天
- 活跃度
- 停滞
- 描述清晰度
- 基本清楚
- 新手友好度
- 45/100