prometheus / prometheus/client_python

Multiproc doesn't capture all metrics

未关闭
#1,089 5 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

还没有人认领这个 Issue。

主要语言
Python
星标
4.4k
派生
876
平均合并
8 天 4 小时
30 天内合并 PR
1

描述

Hey folks,

I have a similar setup to what is described here. The only difference is that I have nginx in front talking to gunicorn over sockets, but I don't think that matters as far as I can tell.

I wasn't able to use this code:

# Using multiprocess collector for registry
def make_metrics_app():
    registry = CollectorRegistry()
    multiprocess.MultiProcessCollector(registry)
    return make_asgi_app(registry=registry)


metrics_app = make_metrics_app()
app.mount("/metrics", metrics_app)

It results in a page that gets a 307, so I implemented this:

@router.get("/metrics", response_class=PlainTextResponse, include_in_schema=False)
async def get_metrics() -> str:
    """
    Get prometheus metrics
    """
    registry = CollectorRegistry()
    multiprocess.MultiProcessCollector(registry)
    data = generate_latest(registry)
    res = Response(content=data)
    res.headers["Content-Type"] = CONTENT_TYPE_LATEST
    res.headers["Content-Length"] = str(len(data))
    return res

Apart from that my setup is pretty typical, I have some counters and a histogram:

AUTOMATIONS_TOTAL: Final = Gauge(
    name=prefix_("automations_total"),
    documentation="Number of automations run",
    labelnames=["platform", "action", "kind", "status"],
    multiprocess_mode="sum",
)

REQUEST_TIME_SECONDS: Final = Histogram(
    name=prefix_("request_time_seconds"),
    documentation="Time spent processing request",
    labelnames=["method", "url_rule", "status_code"],
)

REQUESTS_IN_PROGRESS_TOTAL: Final = Gauge(
    name=prefix_("requests_in_progress_total"),
    documentation="Number of concurrent requests",
    # See Metrics Tuning (Gauge)
    # https://github.com/prometheus/client_python#multiprocess-mode-gunicorn
    multiprocess_mode="sum",
)

My in progress counter always remains at zero, but I'm hoping that's because requests finish faster than I can see them. The histogram, however, never shows up - I see no record of it in my PROMETHEUS_MULTIPROC_DIR and it's never rendered in the output. Any ideas for what I could troubleshoot?

贡献指南

打开贡献指南

从这里开始

  1. 先读完整个 Issue,再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库,在一个分支上完成修改。
  4. 提交 Pull Request,并在描述里引用这个 Issue 编号。

调研方向

首先,复现 issue 中所示的 FastAPI 多进程 collector 路径和自定义 /metrics handler。在比较 counter、gauge 和 histogram 的行为时,检查 CollectorRegistry、MultiProcessCollector 以及 PROMETHEUS_MULTIPROC_DIR 的输出。完成的标准是确定 histogram 缺失的原因,并确认预期的指标已被渲染。

由索引模型根据 Issue 内容生成。

评估

技术栈
fastapi, nginx, prometheus, python
领域
observability
Issue 类型
缺陷
难度
4/5
预计耗时
3-5 天
活跃度
停滞
描述清晰度
需要澄清
新手友好度
35/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。