codalab / codalab/codabench

Data integrity: score or leaderboard write fails after Finished status

Open
#2,423 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
176
Forks
74
Avg merge
5d 3h
Merged PRs (30d)
21

Description

Problem

The compute worker sets submission status to FINISHED before uploading scores and outputs. If push_scores() or push_output() fails (network error, timeout, API failure), the submission is marked Finished but has no scores or outputs.

Root Cause

In compute_worker.py, the flow was:

run.start()         # Sets status to FINISHED at line 1448
if run.is_scoring:
    run.push_scores()  # Upload scores AFTER status update
run.push_output()      # Upload outputs AFTER status update

If push_scores() fails, the submission stays Finished with no scores → data integrity violation.

Impact
  • Users see Finished submissions with no results
  • Leaderboard is incomplete
  • No retry mechanism exists to recover

This bug was discovered during the EEG Foundation Challenge incident analysis.

Solution

  1. Reorder operations: upload scores/outputs before setting status to FINISHED
  2. Add retry logic to push_scores() with exponential backoff
  3. Validate HTTP responses and raise errors on 4xx/5xx
New Flow
run.start()         # Completes scoring but doesn't set FINISHED
if run.is_scoring:
    run.push_scores()       # Upload scores first (with retries)
run.push_output()           # Then upload outputs
if run.is_scoring:
    run._update_status(SubmissionStatus.FINISHED)  # Only now mark as FINISHED

Testing

Added comprehensive K6 integration test:

  • tests/k6/test_m8_finished_has_scores.js — verifies invariant: Finished ⟹ has scores
  • tests/k6/run_m8_test.sh — bash orchestrator
  • Pass criteria: finished_without_scores == 0

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in compute_worker.py by tracing the run.start(), push_scores(), push_output(), and status-update flow described in the issue. Run tests/k6/run_m8_test.sh and use tests/k6/test_m8_finished_has_scores.js to verify that no Finished submission lacks scores; completion also requires the stated retry and HTTP-response behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
bash, python
Domain
backend, data, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.