simonsobs / simonsobs/sorunlib
Check the SMuRF `stream` state when other operations fail
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 0
- Forks
- 0
- Avg merge
- 17h 32m
- Merged PRs (30d)
- 1
Description
We have been seeing a problem when users accidentally leave the SMuRF streams on (whether through user error, or through not realizing an interrupt has left them on -- see #166). This causes all subsequent SMuRF commands, i.e. take_bgmap to fail, as the operations cannot run simultaneously, and so the command can't grab the ocs TimeoutLock, causing it to fail.
In sorunlib this causes us to stop commanding the controller. In practice, initialize() is called again within a few hours, but then another take_bgmap is run, which also fails for the same reason. The result is the stream is never turned off until an observant user notices that this is the reason the controller is being dropped and manually stops them.
The proposed fix for this is to check the SMuRF stream state after an operation fails. If stream is on, turn it off and then drop the slot like usual. If we can't turn it off (up to some timeout period), raise an exception. If stream is off, just drop the slot as normal. We'll still raise an exception if the threshold for number of running slots is crossed.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing initialize(), take_bgmap, and the sorunlib path that handles TimeoutLock failures, including how SMuRF stream state and slots are currently managed. Done means a failed operation checks the stream, stops it when necessary within a timeout, raises if stopping fails, and preserves the existing slot-threshold exception behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100