simonsobs / simonsobs/socs

Crate monitor agent crashes on startup

Open
#894 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug needs triage
Dominant language
Python
Stars
17
Forks
20
Avg merge
1h 22m
Merged PRs (30d)
2

Description

The new v0.5.4 crate monitor agent is crashing on startup on satp3 with:

2025-06-12T14:41:26+0000 startup-op: launching init_crate
2025-06-12T14:41:26+0000 start called for init_crate
2025-06-12T14:41:26+0000 init_crate:0 Status is now "starting".
2025-06-12T14:41:26+0000 init_crate:0 Status is now "running".
2025-06-12T14:41:26+0000 root@shm-crt2-satp3
2025-06-12T14:41:26+0000 command run: ['ssh', 'root@shm-crt2-satp3', 'pwd']
2025-06-12T14:41:26+0000 init_crate:0 CRASH: [Failure instance: Traceback: <class 'IndexError'>: list index out of range
/opt/venv/lib/python3.10/site-packages/twisted/python/threadpool.py:269:inContext
/opt/venv/lib/python3.10/site-packages/twisted/python/threadpool.py:285:<lambda>
/opt/venv/lib/python3.10/site-packages/twisted/python/context.py:117:callWithContext
/opt/venv/lib/python3.10/site-packages/twisted/python/context.py:82:callWithContext
/opt/venv/lib/python3.10/site-packages/ocs/ocs_agent.py:984:_running_wrapper
/opt/venv/lib/python3.10/site-packages/socs/agents/smurf_crate_monitor/agent.py:222:init_crate
]
2025-06-12T14:41:26+0000 init_crate:0 Status is now "done".

v0.5.0-5-g6688b61-dev is the version it was previously running, which would result in:

2025-06-12T14:39:19+0000 startup-op: launching init_crate
2025-06-12T14:39:19+0000 start called for init_crate
2025-06-12T14:39:19+0000 init_crate:0 Status is now "starting".
2025-06-12T14:39:19+0000 init_crate:0 Status is now "running".
2025-06-12T14:39:19+0000 root@shm-crt2-satp3
2025-06-12T14:39:19+0000 command run: ['ssh', 'root@shm-crt2-satp3', 'pwd']
2025-06-12T14:39:20+0000 /etc/home/root

2025-06-12T14:39:20+0000 Successfully ssh-d into shelf
2025-06-12T14:39:20+0000 start called for acq
2025-06-12T14:39:20+0000 acq:1 Status is now "starting".
2025-06-12T14:39:20+0000 acq:1 Status is now "running".
2025-06-12T14:39:20+0000 init_crate:0 Crate Initialized
2025-06-12T14:39:20+0000 init_crate:0 Status is now "done".
2025-06-12T14:39:20+0000 Started acquisition
2025-06-12T14:39:22+0000 Got sensor names

It seems like the ssh connection isn't going through.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in socs/agents/smurf_crate_monitor/agent.py at init_crate, around line 222, and compare the v0.5.4 behavior with v0.5.0-5-g6688b61-dev. Reproduce startup on satp3 and inspect the SSH command and returned output. Done means the agent completes crate initialization and starts acquisition without the IndexError.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.