prometheus / prometheus/client_ruby

`DirectFileStore` creates too many files for long running applications

Open
#143 7 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Ruby
Stars
558
Forks
154
PR merge metrics
No merged PRs in 30d

Description

We use passenger to run our rails application. Worker processes recreated after every 100 requests. Recently we tried client_ruby with DirectFileStore. Each worker process has its own pid so depending on server load there will be tens of thousands of files in prometheus work dir after couple of hours/days when app wasn't restarted.

With 50k files in prometheus work dir /metrics route starts to be very slow with processing time about 10 seconds which could lead to prometheus scraper timeouts.

Is there any workaround for long running processes with DirectFileStore? One possible workaround I can think of is just restart app once in a few hours. Instead of restarting the whole application we can just wipe prometheus work directory once in a few hours, but this is looks a little bit hacky for me.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Read the DirectFileStore implementation and trace how worker PIDs create files in the Prometheus work directory. Reproduce the long-running Passenger scenario and measure the /metrics route with many files. The issue needs a maintainer decision on the intended cleanup or retention behavior; done should prevent unbounded growth without requiring an application restart.

Written by the indexing model from the issue text.

Assessment

Tech stack
ruby
Domain
observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.