True sub-process restart support
- Lingua principale
- Python
- Stelle
- 5.3k
- Fork
- 383
- Merge medio
- 9h 41m
- PR unite (30g)
- 2
Descrizione
Python has a known limitation - it doesn't always return the memory to the OS.
In an environment that have memory constraints like kubernetes, it causes OOM unnecessarily.
To mitigate this, most frameworks restart the process after certain criteria is met: Celery long supports both `worker_max_tasks_per_child` and `worker_max_memory_per_child`, uvicorn has `--limit-max-requests`.
The current suggested [implementation](https://github.com/Bogdanp/dramatiq/blob/master/examples/max_tasks_per_child/app.py) has a major flaw - it restarts all of the sub-processes instead of the specific process that reached the limit.
In multi-process configuration, this causes an unnecessary drainage to the other sub-processes, creating possible latencies until the drain complete. If I were to implement a memory limit middleware, that would be even worse because one sub-process reaching the limit doesn't correlate other sub-processes usage.
Unfortunately, the core of dramatiq doesn't allow this to be implemented: the only support is to restart all of the processes using the `SIGHUP` signal. A previous attempt https://github.com/Bogdanp/dramatiq/pull/236 was rejected, where it didn't address the actual issues.
I would like to fix this by providing an API that will signal a worker restart.
Would appreciate the go-ahead before doing so.
I do have 2 possible implementation in mind: Creating a dedicate middleware exception or using a specific signal (`SIGUSR2`).
Guida per i contributori
Apri la guida per i contributori
Direzione di ricerca
Look at the worker process management in dramatiq/worker.py and the existing signal handling (SIGHUP). Review the rejected PR #236 to understand past issues. The goal is to design an API that allows restarting a single sub-process, possibly via a new middleware exception or a dedicated signal like SIGUSR2. Test by modifying the example at examples/max_tasks_per_child/app.py to ensure only the target process restarts.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Valutazione
- Stack tecnologico
- python, rabbitmq, redis
- Ambito
- backend, distributed-systems
- Tipo di issue
- Funzionalità
- Difficoltà
- 4/5
- Tempo stimato
- 3-5 giorni
- Stato di attività
- Tranquilla
- Chiarezza
- Abbastanza chiara
- Idoneità per principianti
- 45/100