True sub-process restart support
- Langage dominant
- Python
- Étoiles
- 5.3k
- Forks
- 383
- Merge moyen
- 9 h 41 min
- PR mergées (30 j)
- 2
Description
Python has a known limitation - it doesn't always return the memory to the OS.
In an environment that have memory constraints like kubernetes, it causes OOM unnecessarily.
To mitigate this, most frameworks restart the process after certain criteria is met: Celery long supports both `worker_max_tasks_per_child` and `worker_max_memory_per_child`, uvicorn has `--limit-max-requests`.
The current suggested [implementation](https://github.com/Bogdanp/dramatiq/blob/master/examples/max_tasks_per_child/app.py) has a major flaw - it restarts all of the sub-processes instead of the specific process that reached the limit.
In multi-process configuration, this causes an unnecessary drainage to the other sub-processes, creating possible latencies until the drain complete. If I were to implement a memory limit middleware, that would be even worse because one sub-process reaching the limit doesn't correlate other sub-processes usage.
Unfortunately, the core of dramatiq doesn't allow this to be implemented: the only support is to restart all of the processes using the `SIGHUP` signal. A previous attempt https://github.com/Bogdanp/dramatiq/pull/236 was rejected, where it didn't address the actual issues.
I would like to fix this by providing an API that will signal a worker restart.
Would appreciate the go-ahead before doing so.
I do have 2 possible implementation in mind: Creating a dedicate middleware exception or using a specific signal (`SIGUSR2`).
Guide de contribution
Ouvrir le guide de contribution
Piste de recherche
Look at the worker process management in dramatiq/worker.py and the existing signal handling (SIGHUP). Review the rejected PR #236 to understand past issues. The goal is to design an API that allows restarting a single sub-process, possibly via a new middleware exception or a dedicated signal like SIGUSR2. Test by modifying the example at examples/max_tasks_per_child/app.py to ensure only the target process restarts.
Rédigé par le modèle d'indexation à partir du texte de l'issue.
Évaluation
- Stack technique
- python, rabbitmq, redis
- Domaine
- backend, distributed-systems
- Type d'issue
- Fonctionnalité
- Difficulté
- 4/5
- Temps estimé
- 3-5 jours
- Activité
- Calme
- Clarté
- Plutôt claire
- Accessibilité débutants
- 45/100