compute_worker re-executes submissions on broker redelivery, causing duplicate scores, hostname overwrite, and status flipping
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 176
- Forks
- 74
- Avg merge
- 5d 3h
- Merged PRs (30d)
- 21
Description
Issue
When a compute_worker task is interrupted after the work has started but before the message is acked (worker crash, OOM kill, soft time-out, container restart, network partition with the broker), RabbitMQ requeues the same compute_worker_run payload. A subsequent worker pulls it from the broker and runs the full lifecycle a second time end-to-end.
Observed concrete symptoms:
- The submission row's
scoring_worker_hostnamegets overwritten with the second worker's hostname, destroying the audit trail of who actually ran the job. - The submission
statuscycles backwards: a row already inFinishedis PATCHed back throughRunning → Scoring → Finished, breaking downstream signals (notifications, leaderboard recomputation). upload_submission_scoresis called a second time and inserts a duplicateSubmissionScorerow per leaderboard column. The aggregatorSubmission.calculate_scores()then crashes withMultipleObjectsReturnedonsubmission.scores.get(column=col), and any subsequent score read can return either of the duplicates non-deterministically.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the compute_worker_run lifecycle, including where scoring_worker_hostname and submission status are updated. Inspect upload_submission_scores and Submission.calculate_scores() to understand the duplicate-score failure, then reproduce a broker redelivery scenario if the project provides a worker test harness. Done means redelivery does not repeat scoring or regress status and the audit information remains intact.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 45/100