Bogdanp / Bogdanp/dramatiq

True sub-process restart support

Aperta
#882 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub
Lingua principale
Python
Stelle
5.3k
Fork
383
Merge medio
9h 41m
PR unite (30g)
2

Descrizione

Python has a known limitation - it doesn't always return the memory to the OS.
In an environment that have memory constraints like kubernetes, it causes OOM unnecessarily.
To mitigate this, most frameworks restart the process after certain criteria is met: Celery long supports both `worker_max_tasks_per_child` and `worker_max_memory_per_child`, uvicorn has `--limit-max-requests`.
The current suggested [implementation](https://github.com/Bogdanp/dramatiq/blob/master/examples/max_tasks_per_child/app.py) has a major flaw - it restarts all of the sub-processes instead of the specific process that reached the limit.
In multi-process configuration, this causes an unnecessary drainage to the other sub-processes, creating possible latencies until the drain complete. If I were to implement a memory limit middleware, that would be even worse because one sub-process reaching the limit doesn't correlate other sub-processes usage.

Unfortunately, the core of dramatiq doesn't allow this to be implemented: the only support is to restart all of the processes using the `SIGHUP` signal. A previous attempt https://github.com/Bogdanp/dramatiq/pull/236 was rejected, where it didn't address the actual issues.

I would like to fix this by providing an API that will signal a worker restart.
Would appreciate the go-ahead before doing so.

I do have 2 possible implementation in mind: Creating a dedicate middleware exception or using a specific signal (`SIGUSR2`).

Guida per i contributori

Apri la guida per i contributori

Direzione di ricerca

Look at the worker process management in dramatiq/worker.py and the existing signal handling (SIGHUP). Review the rejected PR #236 to understand past issues. The goal is to design an API that allows restarting a single sub-process, possibly via a new middleware exception or a dedicated signal like SIGUSR2. Test by modifying the example at examples/max_tasks_per_child/app.py to ensure only the target process restarts.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
python, rabbitmq, redis
Ambito
backend, distributed-systems
Tipo di issue
Funzionalità
Difficoltà
4/5
Tempo stimato
3-5 giorni
Stato di attività
Tranquilla
Chiarezza
Abbastanza chiara
Idoneità per principianti
45/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.