prometheus / prometheus/client_python

Add usage example for Celery

Open
#902 21 comments 11 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
4.4k
Forks
876
Avg merge
8d 4h
Merged PRs (30d)
1

Description

I recently wanted to collect custom application metrics from a rather large Celery app. I tried several approaches with varying degrees of awkwardness, until finally settling on having a "sidecar" HTTP server process for all Celery nodes.
This works pretty well, but took me a while to get right. To spare others from wasting as much time as me, I would like to propose adding a section on getting Celery metrics set up with the Prometheus client. I wrote up the solution on StackOverflow: https://stackoverflow.com/a/75799358/2532203

If you're interested, I can properly summarise it and open a PR?

Also, if you look at the code, I needed to copy some swaths of the library because the built-in Prometheus WSGI server is hard-coded to demonise the thread; this causes trouble when running in a demonised process. Maybe this could be made optional with another keyword argument that defaults to True?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the StackOverflow solution linked in the issue and the Prometheus client's exposition.py WSGI server reference. Summarize the sidecar HTTP-server setup for collecting metrics from Celery workers in the appropriate documentation section, and verify that the example is complete and understandable. Treat the optional daemon-thread behavior as a separate question unless the existing discussion resolves it.

Written by the indexing model from the issue text.

Assessment

Tech stack
prometheus, python
Domain
documentation, observability-sre
Issue type
Documentation
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.