codalab / codalab/codabench

compute_worker re-executes submissions on broker redelivery, causing duplicate scores, hostname overwrite, and status flipping

Open
#2,433 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
176
Forks
74
Avg merge
5d 3h
Merged PRs (30d)
21

Description

Issue

When a compute_worker task is interrupted after the work has started but before the message is acked (worker crash, OOM kill, soft time-out, container restart, network partition with the broker), RabbitMQ requeues the same compute_worker_run payload. A subsequent worker pulls it from the broker and runs the full lifecycle a second time end-to-end.

Observed concrete symptoms:

  • The submission row's scoring_worker_hostname gets overwritten with the second worker's hostname, destroying the audit trail of who actually ran the job.
  • The submission status cycles backwards: a row already in Finished is PATCHed back through Running → Scoring → Finished, breaking downstream signals (notifications, leaderboard recomputation).
  • upload_submission_scores is called a second time and inserts a duplicate SubmissionScore row per leaderboard column. The aggregator Submission.calculate_scores() then crashes with MultipleObjectsReturned on submission.scores.get(column=col), and any subsequent score read can return either of the duplicates non-deterministically.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the compute_worker_run lifecycle, including where scoring_worker_hostname and submission status are updated. Inspect upload_submission_scores and Submission.calculate_scores() to understand the duplicate-score failure, then reproduce a broker redelivery scenario if the project provides a worker test harness. Done means redelivery does not repeat scoring or regress status and the audit information remains intact.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.