simonsobs / simonsobs/sodetlib

`jackhammer` hangs until controller agents restarted

Open
#463 5 comments 0 reactions 2 assignees View on GitHub

@msilvafe is already working on this.

Since Nov 20, 2025.

bug
Dominant language
Python
Stars
5
Forks
0
Avg merge
3d 17h
Merged PRs (30d)
4

Description

Filing this issue based on reports from @msilvafe and @aashrita because I have a proposed fix.

With jackhammer no longer interacting with the OCS agents, since we split those docker containers into a separate compose file (following the implementation of #451), it seems a jackhammer hammer will hang until the pysmurf-controller agents are rebooted manually by the user using Host Manager.

This was first reported by @msilvafe on June 5th on satp1. @aashrita provided some output from the same issue occurring on satp3 on June 8th. Running the following it hung here until agents were restarted via host manager:

cryo@smurf-so10-satp3:~/docker/pcie/v2.1.1$ jackhammer hammer 4
You are hard-resetting slots [4]. Are you sure (y/n)? y

Dumping docker logs to /data/logs/17494/1749402654
Saving 'docker ps' to /data/logs/17494/1749402654/docker_state.log
Saving ocs-det-controller-c2s5 logs to /data/logs/17494/1749402654/ocs-det-controller-c2s5.log
Saving ocs-det-controller-c2s4 logs to /data/logs/17494/1749402654/ocs-det-controller-c2s4.log
Saving ocs-det-crate-2 logs to /data/logs/17494/1749402654/ocs-det-crate-2.log
Saving ocs-daq-sync-smurf-so10 logs to /data/logs/17494/1749402654/ocs-daq-sync-smurf-so10.log
Saving ocs-det-monitor-so10 logs to /data/logs/17494/1749402654/ocs-det-monitor-so10.log
Proposed Solution

I think the best solution here would be to get jackhammer to resume restarting any required ocs agents, just via the host manager, rather than directly using docker compose itself. This may require adding the host manager instance-id to sys_config.yaml, or maybe just assuming there is only one host manager (which there should be now, in all cases on site).

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.