simonsobs / simonsobs/socs

Cryomech `power_ptc` task crash

Open
#1,004 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

agent: cryomech bug
Dominant language
Python
Stars
17
Forks
20
Avg merge
1h 22m
Merged PRs (30d)
2

Description

Recently on the LAT one of the cryomech agents was seeing a BrokenPipeError while trying to command the power_ptc task:

2026-03-09T01:15:41+0000 start called for acq
2026-03-09T01:15:41+0000 acq:1 Status is now "starting".
2026-03-09T01:15:41+0000 Response from acq.start(): Started process "acq".
2026-03-09T01:15:41+0000 init:0 PTC agent initialized
2026-03-09T01:15:41+0000 init:0 Status is now "done".
2026-03-09T01:15:41+0000 acq:1 Status is now "running".
2026-03-09T01:15:57+0000 stop called for acq
2026-03-09T01:15:57+0000 Stopper for "acq" terminated with ok=True and message ((True, 'requested to stop taking data.'),)
2026-03-09T01:15:58+0000 acq:1 Acquisition exited cleanly.
2026-03-09T01:15:58+0000 acq:1 Status is now "done".
2026-03-09T01:15:59+0000 stop called for acq
2026-03-09T01:16:20+0000 stop called for acq
2026-03-09T01:20:52+0000 start called for power_ptc
2026-03-09T01:20:52+0000 Caught ParamError during start call: Param 'state' is required and must not be None
2026-03-09T01:27:56+0000 start called for power_ptc
2026-03-09T01:27:56+0000 power_ptc:2 Status is now "starting".
2026-03-09T01:27:56+0000 power_ptc:2 Status is now "running".
2026-03-09T01:27:56+0000 power_ptc:2 PTC powered off
2026-03-09T01:27:56+0000 power_ptc:2 Status is now "done".
2026-03-09T01:28:30+0000 start called for power_ptc
2026-03-09T01:28:30+0000 power_ptc:3 Status is now "starting".
2026-03-09T01:28:30+0000 power_ptc:3 Status is now "running".
2026-03-09T01:28:30+0000 power_ptc:3 CRASH: [Failure instance: Traceback: <class 'BrokenPipeError'>: [Errno 32] Broken pipe
/opt/venv/lib/python3.10/site-packages/twisted/python/threadpool.py:269:inContext
/opt/venv/lib/python3.10/site-packages/twisted/python/threadpool.py:285:<lambda>
/opt/venv/lib/python3.10/site-packages/twisted/python/context.py:117:callWithContext
/opt/venv/lib/python3.10/site-packages/twisted/python/context.py:82:callWithContext
/opt/venv/lib/python3.10/site-packages/ocs/ocs_agent.py:984:_running_wrapper
/opt/venv/lib/python3.10/site-packages/socs/agents/cryomech_cpa/agent.py:111:power_ptc
/opt/venv/lib/python3.10/site-packages/socs/agents/cryomech_cpa/drivers.py:95:power
]
2026-03-09T01:28:30+0000 power_ptc:3 Status is now "done".
2026-03-09T01:28:30+0000 start called for power_ptc
2026-03-09T01:28:30+0000 power_ptc:4 Status is now "starting".
2026-03-09T01:28:30+0000 power_ptc:4 Status is now "running".
2026-03-09T01:28:30+0000 power_ptc:4 CRASH: [Failure instance: Traceback: <class 'BrokenPipeError'>: [Errno 32] Broken pipe
/opt/venv/lib/python3.10/site-packages/twisted/python/threadpool.py:269:inContext
/opt/venv/lib/python3.10/site-packages/twisted/python/threadpool.py:285:<lambda>
/opt/venv/lib/python3.10/site-packages/twisted/python/context.py:117:callWithContext
/opt/venv/lib/python3.10/site-packages/twisted/python/context.py:82:callWithContext
/opt/venv/lib/python3.10/site-packages/ocs/ocs_agent.py:984:_running_wrapper
/opt/venv/lib/python3.10/site-packages/socs/agents/cryomech_cpa/agent.py:111:power_ptc
/opt/venv/lib/python3.10/site-packages/socs/agents/cryomech_cpa/drivers.py:95:power
]
2026-03-09T01:28:30+0000 power_ptc:4 Status is now "done".
2026-03-10T12:59:51+0000 start called for power_ptc
2026-03-10T12:59:51+0000 power_ptc:5 Status is now "starting".
2026-03-10T12:59:51+0000 power_ptc:5 Status is now "running".
2026-03-10T12:59:51+0000 power_ptc:5 CRASH: [Failure instance: Traceback: <class 'BrokenPipeError'>: [Errno 32] Broken pipe
/opt/venv/lib/python3.10/site-packages/twisted/python/threadpool.py:269:inContext
/opt/venv/lib/python3.10/site-packages/twisted/python/threadpool.py:285:<lambda>
/opt/venv/lib/python3.10/site-packages/twisted/python/context.py:117:callWithContext
/opt/venv/lib/python3.10/site-packages/twisted/python/context.py:82:callWithContext
/opt/venv/lib/python3.10/site-packages/ocs/ocs_agent.py:984:_running_wrapper
/opt/venv/lib/python3.10/site-packages/socs/agents/cryomech_cpa/agent.py:111:power_ptc
/opt/venv/lib/python3.10/site-packages/socs/agents/cryomech_cpa/drivers.py:95:power
]
2026-03-10T12:59:51+0000 power_ptc:5 Status is now "done".
2026-03-10T14:36:34+0000 start called for acq
2026-03-10T14:36:34+0000 acq:6 Status is now "starting".
2026-03-10T14:36:34+0000 acq:6 Status is now "running".
2026-03-10T14:36:34+0000 Connection error: [Errno 32] Broken pipe
2026-03-10T14:36:34+0000 Resetting the connection to the device.

Seems like the agent lost connection to the device (probably unrelated to the caught ParamError), and couldn't auto-reconnect on subsequent power_ptc calls. It reconnected when acq was restarted, as expected.

socs version: v0.5.8

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with socs/agents/cryomech_cpa/agent.py at power_ptc and socs/agents/cryomech_cpa/drivers.py at power, using the reported BrokenPipeError after the device connection is lost. Reproduce repeated power_ptc calls after disconnecting the device and compare them with restarting acq. Done means the recovery behavior is defined and verified for subsequent power_ptc calls.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.