prometheus / prometheus/client_python

Aggregate expired pid db files, control the number of files and improve scrape effectiveness

Open
#443 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
4.4k
Forks
876
Avg merge
8d 4h
Merged PRs (30d)
1

Description

history related issues: https://github.com/prometheus/client_python/pull/441
https://github.com/prometheus/client_python/pull/430

Can we aggregate all the db files from a period of time ago and non-current pid into a total db file, to control the number of pid files ?

I have realized this idea with golang,here are some details:

Project deploy info:
gunicorn django
128 workers
gunicorn max_requests:10000(create a new pid file almost every minute)

  1. I can't solve the problem that the pid file has been growing, and it can reach 6,000 in four days;
  2. Try to delete the expired pid regularly in the code, but it will cause the figure to drop with grafana;
  3. The time to request metric is getting longer as the program runs.

Improve scrape efficiency:
I used golang to rewrite the logic of python aggregate metrics(generate metric still using python). After rewriting, each scrape time is less than 1 second.

Solve the growing pid files:
Aggregate all the db files from a period of time ago and non-current pid into a total db file. Then delete these files. When calculating metric, history total db + curent pid = current pid db. (i do it every hour)

Now, num of pid files is <200 in my project. if we can do change this, it would be a big strengthen. Just like prometheus will also aggregate historical data

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the history-related pull requests #441 and #430, then trace the Python aggregation and scrape behavior described for the 128-worker Gunicorn deployment. Compare the proposed periodic aggregation and deletion approach with the current metric results in Grafana; done means bounded pid database files, preserved historical values, and scrape time below one second.

Written by the indexing model from the issue text.

Assessment

Tech stack
django, go, grafana, prometheus, python
Domain
backend, observability, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.