prometheus / prometheus/client_python
Duplicated timeseries in CollectorRegistry with Multiprocess Gunicorn
Nadie ha tomado este issue todavía.
- Lenguaje dominante
- Python
- Estrellas
- 4.4k
- Forks
- 876
- Merge medio
- 8 d 4 h
- PR fusionados (30 d)
- 1
Descripción
I know this is a subject that comes up somewhat frequently but for the love of me I can't figure out what I'm doing wrong.
-
I have a service in Amazon ECS thats running a single task with multiple workers (actually the problem happens in my other service that just has one worker also).
-
I've created the directory and set the
PROMETHEUS_MULTIPROC_DIRin the Dockerfile:
RUN mkdir -p /tmp/prom-metrics
ENV PROMETHEUS_MULTIPROC_DIR /tmp/prom-metrics
- I'm using the sample code in the README to create the registry in the
/metricsrequest and return it:
registry = CollectorRegistry()
if getenv('PROMETHEUS_MULTIPROC_DIR'):
multiprocess.MultiProcessCollector(registry)
data = generate_latest(registry)
status = '200 OK'
response_headers = [
('Content-type', CONTENT_TYPE_LATEST),
('Content-Length', str(len(data))),
]
return Response(data, status, response_headers)
- I've created the
gunicorn.conf.pyfile with the sample from the README and passed it into my gunicorn startup script via-c:
from prometheus_client import multiprocess
def child_exit(server, worker):
multiprocess.mark_process_dead(worker.pid)
In my two services, gunicorn starts them as follows:
# app 1 with workers
gunicorn -c /app/utils/gunicorn.conf.py -b :5000 -t 3600 --keep-alive 60 --threads 8 --workers 3 app:app
# app 2 without workers
gunicorn -c /app/utils/gunicorn.conf.py -b :5000 -t 3600 --keep-alive 60 --threads 8 app:app
The service boots successfully and accepts some metrics which are definitely collected in multiprocess mode, seeing as the HELP line simply displays Multiprocess metric.
This works for a few calls but eventually I get the dreaded Duplicated timeseries in CollectorRegistry error and no additional metrics are populated.
What might I be doing wrong?
Guía de contribución
Primeros pasos
- Lee el issue completo y luego la guía de contribución del proyecto.
- Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
- Haz un fork del repositorio y trabaja en una rama.
- Abre un pull request que haga referencia al número del issue.
Línea de trabajo
Comienza con el ejemplo de métricas multiprocess del README y la configuración gunicorn.conf.py referenciada; después, reproduce el fallo usando la configuración de Dockerfile y los comandos de Gunicorn mostrados. El trabajo estará terminado cuando se identifique la causa del error de timeseries duplicadas y se documente o corrija la configuración para que /metrics siga devolviendo métricas.
Escrito por el modelo de indexación a partir del texto del issue.
Evaluación
- Stack tecnológico
- docker, prometheus, python
- Área
- observability
- Tipo de issue
- Error
- Dificultad
- 4/5
- Tiempo estimado
- 3-5 días
- Estado de actividad
- Estancado
- Claridad
- Necesita aclaración
- Aptitud para principiantes
- 25/100