prometheus / prometheus/client_ruby
`DirectFileStore` creates too many files for long running applications
Nobody has claimed this yet.
- Dominant language
- Ruby
- Stars
- 558
- Forks
- 154
- PR merge metrics
- No merged PRs in 30d
Description
We use passenger to run our rails application. Worker processes recreated after every 100 requests. Recently we tried client_ruby with DirectFileStore. Each worker process has its own pid so depending on server load there will be tens of thousands of files in prometheus work dir after couple of hours/days when app wasn't restarted.
With 50k files in prometheus work dir /metrics route starts to be very slow with processing time about 10 seconds which could lead to prometheus scraper timeouts.
Is there any workaround for long running processes with DirectFileStore? One possible workaround I can think of is just restart app once in a few hours. Instead of restarting the whole application we can just wipe prometheus work directory once in a few hours, but this is looks a little bit hacky for me.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Read the DirectFileStore implementation and trace how worker PIDs create files in the Prometheus work directory. Reproduce the long-running Passenger scenario and measure the /metrics route with many files. The issue needs a maintainer decision on the intended cleanup or retention behavior; done should prevent unbounded growth without requiring an application restart.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- ruby
- Domain
- observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100