Compute worker issues

Hopefully solving several points: #2223

1. Containers not removed

• 11/02/2026: submissions containers staying up forever (may be solved by #2223)

EDIT: still happening with MOT20 challenge:

#2343

2. Wrong log when storage is full

When docker pull fails because of full storage, we have no clear logs.
See:

• #2206

• #2217

• Have the right error logs, and have them on the platform's UI

• Detect errors in _get_container_image()

Then it gets stuck in Running state. Solved by #2223.

3. Progress bar

Related: show_progress and the progress bar adds up to the mess:

• Make show_progress() more robust (not treating missing keys as errors)

• Avoid printing the multiple error lines like this (#2223):

2026-02-28 02:38:37.854 | ERROR    | compute_worker:show_progress:137 - There was an error showing the progress bar
2026-02-28 02:38:37.854 | ERROR    | compute_worker:show_progress:138 - 6
2026-02-28 02:38:37.955 | ERROR    | compute_worker:show_progress:137 - There was an error showing the progress bar
2026-02-28 02:38:37.955 | ERROR    | compute_worker:show_progress:138 - 1

4. Logs

• Sometimes no submission logs (#2223)
• Add logs at the start of submission container with metadata of the competition and submission
• Add a clear log in the computer worker container with the competition title when receiving a submission
• Similarly to other problems reported, sometimes we only have "Time limit exceeded" and no other logs (e.g. #1994) (#2223)
• Docker pull and progress bar should be shown during preparation in live logs:

5. No space left

How to manage the disks? Should we limit docker images size?

We could run a prune when docker pull hits the storage limit:

• #2275

6. Option for container shared memory

• Add shm-size as a compute_worker .env setting

More details here:

• #1189

7. Submissions not marked as Failed

Submissions stuck in "Running" or "Scoring" or status

• Submissions stuck in "Scoring" state instead of "Failed" when the compute worker crashes (#2030, #2223)

Related issues:

• #2258 (grouped issue)

• #1203

• #1184

• #1257

• #1821

• #1994

• #2169

• #2177

• Similarly, it looks like the status get stuck to "Preparing" when failing during this process.

Example failure during "Preparing":

[2025-09-18 11:25:05,234: ERROR/ForkPoolWorker-2] Task compute_worker_run[fd956bf5-3e2d-4168-ab48-f0896dc80993] raised unexpected: OSError(28, 'No space left on device')
Traceback (most recent call last):
[...]
OSError: [Errno 28] No space left on device

8. Duplication of submission files

• #1874 (#2285)

9. Scoring and ingestion only works without directory structure

Classic CodaLab and Codabench bug, if the scoring program or ingestion program is inside a folder in the zip, the submission fails.

We need either to:

• Add clear logs when it is the case
• Make it robust so it does not fail

Related issues:

• #968
• #2283

Tentative fix #1905 got reversed by #1946.

10. To check: log level

The log level is defined in this way in compute_worker.py:

configure_logging(
    os.environ.get("LOG_LEVEL", "INFO"), os.environ.get("SERIALIZED", "false")
)

Generally we want as much log as possible, so we may want to be in "DEBUG" log level.

Related:

• #2294

11. Docker pull failing

• Docker pull failing

Pull for image: codalab/codalab-legacy:py39 returned a non-zero exit code! Check if the docker image exists on docker hub.

Related issues:

• #1184
• #1278
• #1263

Solution:

• To have more logs, we need to update compute_worker.py so we print more logs in the logger (#1283).

12. Logs at the wrong place

• Docker pull error during scoring are written in ingestion stderr instead of scoring stdrerr (#1204)
  Solved by: #1214

13. No hostname in server status when status is "Preparing"

• The "Preparing" status means that the worker is downloading the necessary data and programs to run the submission. We should already have a hostname in the server status page during this phase, but it is not the case. (fixed in https://github.com/codalab/codabench/pull/2030)

https://www.codabench.org/server_status

Start with compute_worker.py and the remaining unchecked items in this tracking issue, then read the referenced issues such as #2223, #2275, #1189, #1874, and #2283. The scope spans container cleanup, logging, disk limits, shared memory, and submission handling; done requires resolving the selected outstanding problems and confirming the related worker behavior.