prometheus / prometheus/client_python

Lack of PID uniqueness in container environments makes prometheus_multiproc difficult to use

Open
#823 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
4.4k
Forks
876
Avg merge
8d 4h
Merged PRs (30d)
1

Description

In environments where each container is running in its own PID namespace, multiple instances of the same container will quickly result in (silently) conflicting file names as any my-favorite-server.py process will likely have the same low, single or double digit PID in each of the multiple containers. As a result, the different processes will attempt to mmap the same file in the shared multiprocess metric directory, and various scary errors that aren't always easy to understand the root cause will occur.

It would be beneficial if something in addition to the PID was used to help ensure uniqueness across the files in the PROMETHEUS_MULTIPROC_DIR, or if a string could be passed in at initialization time to be embedded in the file names, such as a container or pod name, or another value determined by the application.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing how prometheus_multiproc names files in PROMETHEUS_MULTIPROC_DIR and how it derives names from process IDs. Compare behavior across multiple containers sharing the directory, then determine a uniqueness mechanism or initialization value that fits the issue. Done means concurrent container processes no longer mmap conflicting files and the relevant failure case is covered by tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
prometheus, python
Domain
observability
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.