Bogdanp / Bogdanp/django_dramatiq
Race condition at AdminMiddleware
- Dominant language
- Python
- Stars
- 384
- Forks
- 86
- PR merge metrics
- No merged PRs in 30d
Description
Hi. We have a lot of tasks in ENQUEUED status but they executed and jobs well done.
I think we have small race condition at middleware callbacks:
1. `Scheduler` emit before_enqueue
2. `Scheduler` publish task to queue
3. `Worker` got task from queue
4. `Worker` emit before processing (update task status=RUNNING)
5. `Worker` done task
6. `Worker` emit after processing. (update task status=DONE)
7. `Scheduler` emit afer_enqueue (update task status=ENQUEUED)
In logs this looks like this:
You can see different order for `Create` and `Update` logs. But last log is `Update`. Do not worry, message logged before SQL operation and this is not equal reflection of SQL updates.
---
So, i want make a PR to fix that. I see two solutions for that case:
1. Optimistic locks: check row version at update and ignore if it changed (this require schema update)
2. Do not allow change status from DONE/RUNNING to ENQUEUED. Check status when update. (control state flow)
@amureki what do you think about that?
Contributor guide
Assessment
This issue has not been assessed yet.