Bogdanp / Bogdanp/django_dramatiq

Race condition at AdminMiddleware

Open
#154 7 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
384
Forks
86
PR merge metrics
No merged PRs in 30d

Description

Hi. We have a lot of tasks in ENQUEUED status but they executed and jobs well done.

I think we have small race condition at middleware callbacks:

1. `Scheduler` emit before_enqueue
2. `Scheduler` publish task to queue
3. `Worker` got task from queue
4. `Worker` emit before processing (update task status=RUNNING)
5. `Worker` done task
6. `Worker` emit after processing. (update task status=DONE)
7. `Scheduler` emit afer_enqueue (update task status=ENQUEUED)

In logs this looks like this:
image

You can see different order for `Create` and `Update` logs. But last log is `Update`. Do not worry, message logged before SQL operation and this is not equal reflection of SQL updates.

---

So, i want make a PR to fix that. I see two solutions for that case:
1. Optimistic locks: check row version at update and ignore if it changed (this require schema update)
2. Do not allow change status from DONE/RUNNING to ENQUEUED. Check status when update. (control state flow)

@amureki what do you think about that?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.