prometheus / prometheus/client_python

gauge.set_function() doesn't work in multiprocess mode

Open
#504 6 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
4.4k
Forks
876
Avg merge
8d 4h
Merged PRs (30d)
1

Description

Multiprocess mode's collect() reads the registry files and aggregates metrics that have been written to prometheus_multiproc_dir.

This doesn't work with gauge.set_function() which does not record its value. The provided function is just called during collection.

That means with the following code:

registry = CollectorRegistry()
Gauge("test", "test", registry=self._registry).set_function(lambda: 100)
multiprocess.MultiProcessCollector(registry)

The output will be:

# HELP test test
# TYPE test gauge
test 100.0
# HELP test Multiprocess metric
# TYPE test gauge
test{pid="10705"} 0.0

Current side effects:

  • If any mode other than all or liveall is used, the pid tag won't be included. This results in duplicate metrics being reported to Prometheus. Prometheus currently only uses the first metric it reads, which is non-deterministic due to iteration over the registry's dictionary.
  • If registry=None to avoid double reporting, only the default value of 0.0 is reported.
  • Current way to work around it is to use the mode all and to ignore gauges in Prometheus and with the tag pid.

Proposal:
I'm not sure how you could incorporate set_function into the multiprocess registry and I'm not convinced how useful of a feature it would be. Is it reasonable to add a new multiprocess_mode: exclude which would prevent the incorrect 0.0 value being reported? Or would it be better to just add documentation to recommend using two independent registries?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the discrepancy using multiprocess.py's collect() entry point and the registry.py set_function() behavior, with prometheus_multiproc_dir configured. Compare the reported values across multiprocess modes and determine the expected handling for set_function(); done means the chosen behavior is documented and the duplicate or incorrect 0.0 reporting is addressed.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
observability-sre
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.