simonsobs / simonsobs/ocs

Error pulling Docker images via HostManager

Open
#479 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug needs triage
Dominant language
Python
Stars
19
Forks
7
PR merge metrics
No merged PRs in 30d

Description

In the first on site usage of the new HM docker_pull task I saw it pull fine sometimes, but also sometimes give:

May 14 17:13:49 daq-satp1 launcher-daq-hm-daq-satp1.sh[128861]: 2026-05-14T17:13:49+0000 manager:0 Requested launch for FTSAerotechAgent:fts-uchicago-so
May 14 17:14:08 daq-satp1 launcher-daq-hm-daq-satp1.sh[128861]: 2026-05-14T17:14:08+0000 manager:0 Launch not detected for FTSAerotechAgent:fts-uchicago-so! Will retry.
May 14 17:14:10 daq-satp1 launcher-daq-hm-daq-satp1.sh[128861]: 2026-05-14T17:14:10+0000 start called for docker_pull
May 14 17:14:10 daq-satp1 launcher-daq-hm-daq-satp1.sh[128861]: 2026-05-14T17:14:10+0000 docker_pull:1 Status is now "starting".
May 14 17:14:10 daq-satp1 launcher-daq-hm-daq-satp1.sh[128861]: 2026-05-14T17:14:10+0000 docker_pull:1 Status is now "running".
May 14 17:14:10 daq-satp1 launcher-daq-hm-daq-satp1.sh[128861]: 2026-05-14T17:14:10+0000 docker_pull:1 Running pull for /home/ocs/config/satp1/daq-satp1/docker-compose.yml ...
May 14 17:14:14 daq-satp1 launcher-daq-hm-daq-satp1.sh[128861]: 2026-05-14T17:14:14+0000 manager:0 Requested launch for FTSAerotechAgent:fts-uchicago-so
May 14 17:14:30 daq-satp1 launcher-daq-hm-daq-satp1.sh[128861]: 2026-05-14T17:14:30+0000 manager:0 Launch not detected for FTSAerotechAgent:fts-uchicago-so! Will retry.
May 14 17:14:30 daq-satp1 launcher-daq-hm-daq-satp1.sh[128861]: 2026-05-14T17:14:30+0000 docker_pull:1 CRASH: [Failure instance: Traceback: <class 'twisted.internet.defer.CancelledError'>:
May 14 17:14:30 daq-satp1 launcher-daq-hm-daq-satp1.sh[128861]: --- <exception caught here> ---
May 14 17:14:30 daq-satp1 launcher-daq-hm-daq-satp1.sh[128861]: /home/ocs/.venv/lib/python3.10/site-packages/ocs/agents/host_manager/agent.py:726:docker_pull
May 14 17:14:30 daq-satp1 launcher-daq-hm-daq-satp1.sh[128861]: ]
May 14 17:14:30 daq-satp1 launcher-daq-hm-daq-satp1.sh[128861]: 2026-05-14T17:14:30+0000 docker_pull:1 Status is now "done".
May 14 17:14:36 daq-satp1 launcher-daq-hm-daq-satp1.sh[128861]: 2026-05-14T17:14:36+0000 manager:0 Requested launch for FTSAerotechAgent:fts-uchicago-so
May 14 17:14:51 daq-satp1 launcher-daq-hm-daq-satp1.sh[128861]: 2026-05-14T17:14:51+0000 manager:0 Launch not detected for FTSAerotechAgent:fts-uchicago-so! Will retry.

I did trigger pulls on several machines at the same time. I wonder if that caused slow downloads and some kind of timeout.

EDIT: Running the task a second time succeeds almost immediately.

ocs version: v0.12.1

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at ocs/agents/host_manager/agent.py:726, where the docker_pull task reports the CancelledError, and reproduce simultaneous pulls on several machines. Compare the first and second task runs while checking whether concurrent downloads trigger cancellation. Done means the initial pull completes reliably without the intermittent cancellation during concurrent use.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, python
Domain
devops
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.