apache / apache/cloudstack

Scrape duration using prometheus is increasing continuously until restart

Open
#13,586 2 comments 0 reactions 0 assignees Claimed by @DaanHoogland View on GitHub
component:prometheus
Dominant language
Java
Stars
3.1k
Forks
1.4k
Avg merge
6d 19h
Merged PRs (30d)
32

Description

### problem

We are using Prometheus to scrape metrics from the management server.
We see a issue that the scrape_duration_seconds metric for the cloudstack job is increasing in slow pace and after ~7-10 days the scrape duration is longer then the configured interval and we get no more metrics.
First time we saw this just doing a time curl towards the mangement server it took 40-50s for metrics to come back.

Image

### versions

CloudStack 4.22.0.0

### The steps to reproduce the bug

1. Start scraping
2. See the scrape_duration_seconds{job="cloudstack"} increasing daily.

### What to do about it?

1. Restart the management server solves the issue, temporarily

Contributor guide

Open the contributing guide

Research direction

Start at the CloudStack management server's Prometheus metrics endpoint and reproduce the issue by scraping it over several days while watching scrape_duration_seconds{job="cloudstack"}. Compare the endpoint response time with the configured scrape interval; done means scrape duration no longer increases continuously and metrics remain available without restarting the management server.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, prometheus
Domain
backend, observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.