Scrape duration using prometheus is increasing continuously until restart
- Dominant language
- Java
- Stars
- 3.1k
- Forks
- 1.4k
- Avg merge
- 6d 19h
- Merged PRs (30d)
- 32
Description
### problem
We are using Prometheus to scrape metrics from the management server.
We see a issue that the scrape_duration_seconds metric for the cloudstack job is increasing in slow pace and after ~7-10 days the scrape duration is longer then the configured interval and we get no more metrics.
First time we saw this just doing a time curl towards the mangement server it took 40-50s for metrics to come back.
### versions
CloudStack 4.22.0.0
### The steps to reproduce the bug
1. Start scraping
2. See the scrape_duration_seconds{job="cloudstack"} increasing daily.
### What to do about it?
1. Restart the management server solves the issue, temporarily
Contributor guide
Research direction
Start at the CloudStack management server's Prometheus metrics endpoint and reproduce the issue by scraping it over several days while watching scrape_duration_seconds{job="cloudstack"}. Compare the endpoint response time with the configured scrape interval; done means scrape duration no longer increases continuously and metrics remain available without restarting the management server.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, prometheus
- Domain
- backend, observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100