simonsobs / simonsobs/sorunlib

Check the SMuRF `stream` state when other operations fail

Open
#229 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Python
Stars
0
Forks
0
Avg merge
17h 32m
Merged PRs (30d)
1

Description

We have been seeing a problem when users accidentally leave the SMuRF streams on (whether through user error, or through not realizing an interrupt has left them on -- see #166). This causes all subsequent SMuRF commands, i.e. take_bgmap to fail, as the operations cannot run simultaneously, and so the command can't grab the ocs TimeoutLock, causing it to fail.

In sorunlib this causes us to stop commanding the controller. In practice, initialize() is called again within a few hours, but then another take_bgmap is run, which also fails for the same reason. The result is the stream is never turned off until an observant user notices that this is the reason the controller is being dropped and manually stops them.

The proposed fix for this is to check the SMuRF stream state after an operation fails. If stream is on, turn it off and then drop the slot like usual. If we can't turn it off (up to some timeout period), raise an exception. If stream is off, just drop the slot as normal. We'll still raise an exception if the threshold for number of running slots is crossed.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing initialize(), take_bgmap, and the sorunlib path that handles TimeoutLock failures, including how SMuRF stream state and slots are currently managed. Done means a failed operation checks the stream, stops it when necessary within a timeout, raises if stopping fails, and preserves the existing slot-threshold exception behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.