Error is not clear after reaching the computation limit of a submission
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 176
- Forks
- 74
- Avg merge
- 5d 3h
- Merged PRs (30d)
- 21
Description
Hello,
It appears that when a submission reaches the computation budget limit, the error associated with the failure is not clear, and the logs of the submission cannot be shown. Here is an example with a specific submission (913840) :
No scoring output:
Then i inspected the logs from the codabench worker, and I saw that it reaches the computation time limit:
For such expected case, the error should be clearer. Also, it would be ideal if log updates were robust to worker interrupts.
Thanks.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the computation-limit failure using submission 913840 and begin with the Codabench worker logs referenced in the issue. Trace how the worker reports the limit and how submission logs are updated after an interrupt; done means the user sees a clear expected-limit error and log updates remain available after worker interruption.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100