Error pulling Docker images via HostManager
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 19
- Forks
- 7
- PR merge metrics
- No merged PRs in 30d
Description
In the first on site usage of the new HM docker_pull task I saw it pull fine sometimes, but also sometimes give:
May 14 17:13:49 daq-satp1 launcher-daq-hm-daq-satp1.sh[128861]: 2026-05-14T17:13:49+0000 manager:0 Requested launch for FTSAerotechAgent:fts-uchicago-so
May 14 17:14:08 daq-satp1 launcher-daq-hm-daq-satp1.sh[128861]: 2026-05-14T17:14:08+0000 manager:0 Launch not detected for FTSAerotechAgent:fts-uchicago-so! Will retry.
May 14 17:14:10 daq-satp1 launcher-daq-hm-daq-satp1.sh[128861]: 2026-05-14T17:14:10+0000 start called for docker_pull
May 14 17:14:10 daq-satp1 launcher-daq-hm-daq-satp1.sh[128861]: 2026-05-14T17:14:10+0000 docker_pull:1 Status is now "starting".
May 14 17:14:10 daq-satp1 launcher-daq-hm-daq-satp1.sh[128861]: 2026-05-14T17:14:10+0000 docker_pull:1 Status is now "running".
May 14 17:14:10 daq-satp1 launcher-daq-hm-daq-satp1.sh[128861]: 2026-05-14T17:14:10+0000 docker_pull:1 Running pull for /home/ocs/config/satp1/daq-satp1/docker-compose.yml ...
May 14 17:14:14 daq-satp1 launcher-daq-hm-daq-satp1.sh[128861]: 2026-05-14T17:14:14+0000 manager:0 Requested launch for FTSAerotechAgent:fts-uchicago-so
May 14 17:14:30 daq-satp1 launcher-daq-hm-daq-satp1.sh[128861]: 2026-05-14T17:14:30+0000 manager:0 Launch not detected for FTSAerotechAgent:fts-uchicago-so! Will retry.
May 14 17:14:30 daq-satp1 launcher-daq-hm-daq-satp1.sh[128861]: 2026-05-14T17:14:30+0000 docker_pull:1 CRASH: [Failure instance: Traceback: <class 'twisted.internet.defer.CancelledError'>:
May 14 17:14:30 daq-satp1 launcher-daq-hm-daq-satp1.sh[128861]: --- <exception caught here> ---
May 14 17:14:30 daq-satp1 launcher-daq-hm-daq-satp1.sh[128861]: /home/ocs/.venv/lib/python3.10/site-packages/ocs/agents/host_manager/agent.py:726:docker_pull
May 14 17:14:30 daq-satp1 launcher-daq-hm-daq-satp1.sh[128861]: ]
May 14 17:14:30 daq-satp1 launcher-daq-hm-daq-satp1.sh[128861]: 2026-05-14T17:14:30+0000 docker_pull:1 Status is now "done".
May 14 17:14:36 daq-satp1 launcher-daq-hm-daq-satp1.sh[128861]: 2026-05-14T17:14:36+0000 manager:0 Requested launch for FTSAerotechAgent:fts-uchicago-so
May 14 17:14:51 daq-satp1 launcher-daq-hm-daq-satp1.sh[128861]: 2026-05-14T17:14:51+0000 manager:0 Launch not detected for FTSAerotechAgent:fts-uchicago-so! Will retry.
I did trigger pulls on several machines at the same time. I wonder if that caused slow downloads and some kind of timeout.
EDIT: Running the task a second time succeeds almost immediately.
ocs version: v0.12.1
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at ocs/agents/host_manager/agent.py:726, where the docker_pull task reports the CancelledError, and reproduce simultaneous pulls on several machines. Compare the first and second task runs while checking whether concurrent downloads trigger cancellation. Done means the initial pull completes reliably without the intermittent cancellation during concurrent use.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, python
- Domain
- devops
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100