codalab / codalab/codabench

_update_submission() has no retry and _update_status() silently swallows exceptions, causing permanent submission loss

Open
#2,415 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
176
Forks
74
Avg merge
5d 3h
Merged PRs (30d)
21

Description

Bug

_update_submission() in compute_worker/compute_worker.py performs a single HTTP PATCH to report submission status back to Django. On any non-200 response, it raises SubmissionException immediately with no retry:

resp = self.requests_session.patch(url, data=data, timeout=150)
if resp.status_code == 200:
    logger.info("Submission updated successfully!")
else:
    raise SubmissionException("Failure updating submission data.")

The caller _update_status() then catches all exceptions and silently discards them:

try:
    self._update_submission(data)
except Exception as e:
    # Always catch exception and never raise error
    logger.exception(f"Failed to update submission status to {status}: {e}")

This means a single transient network error (502, timeout, connection reset) permanently loses the submission result. The compute worker moves on to the next job as if nothing happened, while Django never receives the status=Finished PATCH. The submission stays stuck in Scoring or Running forever.

How it happens

  1. Compute worker finishes scoring and sends PATCH /api/submissions/{id}/ {status: Finished}
  2. The network returns a transient error (502, timeout, connection reset)
  3. _update_submission() raises SubmissionException("Failure updating submission data.")
  4. _update_status() catches the exception, logs it, and does not re-raise
  5. The compute worker proceeds to the next submission
  6. Django never receives the Finished status → the submission stays stuck in Scoring indefinitely
  7. The participant sees a perpetually "processing" submission with no error message and no way to know it failed

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in compute_worker/compute_worker.py by reading _update_submission() and its caller _update_status(), then trace how PATCH failures are handled. Done means transient reporting failures no longer permanently lose the submission result and the worker does not silently discard an unsuccessful status update.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.