Scrape duration using prometheus is increasing continuously until restart
- 主要语言
- Java
- 星标
- 3.1k
- 派生
- 1.4k
- 平均合并
- 6 天 19 小时
- 30 天内合并 PR
- 32
描述
### problem
We are using Prometheus to scrape metrics from the management server.
We see a issue that the scrape_duration_seconds metric for the cloudstack job is increasing in slow pace and after ~7-10 days the scrape duration is longer then the configured interval and we get no more metrics.
First time we saw this just doing a time curl towards the mangement server it took 40-50s for metrics to come back.
### versions
CloudStack 4.22.0.0
### The steps to reproduce the bug
1. Start scraping
2. See the scrape_duration_seconds{job="cloudstack"} increasing daily.
### What to do about it?
1. Restart the management server solves the issue, temporarily
贡献指南
调研方向
从 CloudStack 管理服务器的 Prometheus metrics endpoint 开始,通过连续数天对其进行抓取来重现该问题,同时观察 scrape_duration_seconds{job="cloudstack"}。将 endpoint 响应时间与配置的抓取间隔进行比较;当抓取持续时间不再持续增加,并且 metrics 在无需重启管理服务器的情况下仍然可用时,即表示完成。
由索引模型根据 Issue 内容生成。
评估
- 技术栈
- java, prometheus
- 领域
- backend, observability
- Issue 类型
- 缺陷
- 难度
- 4/5
- 预计耗时
- 3-5 天
- 活跃度
- 停滞
- 描述清晰度
- 基本清楚
- 新手友好度
- 25/100