DiamondLightSource / DiamondLightSource/mx-bluesky

SampleHandlingCallback doesn't register errors in contingency handlers if cleanups are not waited on

Open
#1,369 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
4
Forks
5
Avg merge
1d 30m
Merged PRs (30d)
2

Description

If an error occurs from an action in a finalize_decorator, if it isn't waited on, then it won't be recorded as such.

During overnight collection running mx-bluesky 1.5.9, the gridscan parameters did not validate, which caused a SampleException to be raised. This, as intended did not cause UDC to be interrupted, and was intercepted by the `SampleHandlingCallback` and correctly handled.

```
ESC[32m[I 251014 18:21:42.506 ispyb_store:129] End ispyb deposition with status 'fail' and reason 'Scan invalid - gridscan not valid for detected pin position'.ESC[0m
ESC[32m[I 251014 18:21:45.251 sample_handling_callback:53] Sample handling callback intercepted exception of type SampleException: Scan invalid - gridscan not valid for detected pin positionESC[0m
```

However subsequently, the tidyup in `_overall_tidy()` failed
```
ESC[31m[E 251014 18:21:45.257 run_engine:1755] Run abortedESC[0m
Traceback (most recent call last):
File "/dls_sw/apps/python/miniforge/4.10.0-0/envs/python3.11/lib/python3.11/asyncio/tasks.py", line 490, in wait_for
return fut.result()
^^^^^^^^^^^^
File "/dls_sw/i03/software/bluesky/mx-bluesky_v1.5.9/mx-bluesky/.venv/lib/python3.11/site-packages/ophyd_async/core/_signal.py", line 536, in _wait_for_value
async for value in observe_value(signal):
File "/dls_sw/i03/software/bluesky/mx-bluesky_v1.5.9/mx-bluesky/.venv/lib/python3.11/site-packages/ophyd_async/core/_signal.py", line 428, in observe_value
async for _, value in observe_signals_value(
File "/dls_sw/i03/software/bluesky/mx-bluesky_v1.5.9/mx-bluesky/.venv/lib/python3.11/site-packages/ophyd_async/core/_signal.py", line 509, in observe_signals_value
item = await asyncio.wait_for(q.get(), iteration_timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/dls_sw/apps/python/miniforge/4.10.0-0/envs/python3.11/lib/python3.11/asyncio/tasks.py", line 442, in wait_for
return await fut
^^^^^^^^^
File "/dls_sw/apps/python/miniforge/4.10.0-0/envs/python3.11/lib/python3.11/asyncio/queues.py", line 158, in get
await getter
asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/dls_sw/i03/software/bluesky/mx-bluesky_v1.5.9/mx-bluesky/.venv/lib/python3.11/site-packages/ophyd_async/core/_signal.py", line 545, in wait_for_value
await asyncio.wait_for(self._wait_for_value(signal), timeout)
File "/dls_sw/apps/python/miniforge/4.10.0-0/envs/python3.11/lib/python3.11/asyncio/tasks.py", line 492, in wait_for
raise exceptions.TimeoutError() from exc
TimeoutError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/dls_sw/i03/software/bluesky/mx-bluesky_v1.5.9/mx-bluesky/.venv/lib/python3.11/site-packages/ophyd_async/core/_detector.py", line 261, in unstage
await asyncio.gather(self._writer.close(), self._controller.disarm())
File "/dls_sw/i03/software/bluesky/mx-bluesky_v1.5.9/mx-bluesky/.venv/lib/python3.11/site-packages/ophyd_async/fastcs/panda/_control.py", line 42, in disarm
await wait_for_value(self.pcap.active, False, timeout=1)
File "/dls_sw/i03/software/bluesky/mx-bluesky_v1.5.9/mx-bluesky/.venv/lib/python3.11/site-packages/ophyd_async/core/_signal.py", line 579, in wait_for_value
await checker.wait_for_value(signal, timeout)
File "/dls_sw/i03/software/bluesky/mx-bluesky_v1.5.9/mx-bluesky/.venv/lib/python3.11/site-packages/ophyd_async/core/_signal.py", line 547, in wait_for_value
raise TimeoutError(
TimeoutError: panda-pcap-active didn't match False in 1s, last value None

```

## Acceptance Criteria
- Specific criteria that will be used to judge if the issue is fixed

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.