DiamondLightSource / DiamondLightSource/dodal

Flakey gridscan with zebra test

Open
#2,179 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
5
Forks
13
Avg merge
3d 2h
Merged PRs (30d)
10

Description

https://github.com/DiamondLightSource/dodal/actions/runs/32122592740/job/95666087826?pr=2121

The following test occasionally fails with:

```
=================================== FAILURES ===================================
__________ test_gridscan_prepare_times_out_for_validity_check[zebra] ___________
Traceback (most recent call last):
File "/home/runner/work/dodal/dodal/.venv/lib/python3.11/site-packages/_pytest/runner.py", line 353, in from_call
result: TResult | None = func()
^^^^^^
File "/home/runner/work/dodal/dodal/.venv/lib/python3.11/site-packages/_pytest/runner.py", line 245, in
lambda: runtest_hook(item=item, **kwds),
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/runner/work/dodal/dodal/.venv/lib/python3.11/site-packages/pluggy/_hooks.py", line 512, in __call__
return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/runner/work/dodal/dodal/.venv/lib/python3.11/site-packages/pluggy/_manager.py", line 120, in _hookexec
return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/runner/work/dodal/dodal/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 167, in _multicall
raise exception
File "/home/runner/work/dodal/dodal/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 139, in _multicall
teardown.throw(exception)
File "/home/runner/work/dodal/dodal/.venv/lib/python3.11/site-packages/_pytest/logging.py", line 850, in pytest_runtest_call
yield
File "/home/runner/work/dodal/dodal/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 139, in _multicall
teardown.throw(exception)
File "/home/runner/work/dodal/dodal/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 53, in run_old_style_hookwrapper
return result.get_result()
^^^^^^^^^^^^^^^^^^^
File "/home/runner/work/dodal/dodal/.venv/lib/python3.11/site-packages/pluggy/_result.py", line 103, in get_result
raise exc.with_traceback(tb)
File "/home/runner/work/dodal/dodal/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 38, in run_old_style_hookwrapper
res = yield
^^^^^
File "/home/runner/work/dodal/dodal/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 139, in _multicall
teardown.throw(exception)
File "/home/runner/work/dodal/dodal/.venv/lib/python3.11/site-packages/_pytest/capture.py", line 900, in pytest_runtest_call
return (yield)
^^^^^
File "/home/runner/work/dodal/dodal/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 139, in _multicall
teardown.throw(exception)
File "/home/runner/work/dodal/dodal/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 53, in run_old_style_hookwrapper
return result.get_result()
^^^^^^^^^^^^^^^^^^^
File "/home/runner/work/dodal/dodal/.venv/lib/python3.11/site-packages/pluggy/_result.py", line 103, in get_result
raise exc.with_traceback(tb)
File "/home/runner/work/dodal/dodal/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 38, in run_old_style_hookwrapper
res = yield
^^^^^
File "/home/runner/work/dodal/dodal/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 139, in _multicall
teardown.throw(exception)
File "/home/runner/work/dodal/dodal/.venv/lib/python3.11/site-packages/_pytest/skipping.py", line 268, in pytest_runtest_call
return (yield)
^^^^^
File "/home/runner/work/dodal/dodal/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 121, in _multicall
res = hook_impl.function(*args)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/runner/work/dodal/dodal/.venv/lib/python3.11/site-packages/_pytest/runner.py", line 179, in pytest_runtest_call
item.runtest()
File "/home/runner/work/dodal/dodal/.venv/lib/python3.11/site-packages/pytest_asyncio/plugin.py", line 469, in runtest
super().runtest()
File "/home/runner/work/dodal/dodal/.venv/lib/python3.11/site-packages/_pytest/python.py", line 1720, in runtest
self.ihook.pytest_pyfunc_call(pyfuncitem=self)
File "/home/runner/work/dodal/dodal/.venv/lib/python3.11/site-packages/pluggy/_hooks.py", line 512, in __call__
return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/runner/work/dodal/dodal/.venv/lib/python3.11/site-packages/pluggy/_manager.py", line 120, in _hookexec
return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/runner/work/dodal/dodal/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 167, in _multicall
raise exception
File "/home/runner/work/dodal/dodal/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 139, in _multicall
teardown.throw(exception)
File "/home/runner/work/dodal/dodal/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 53, in run_old_style_hookwrapper
return result.get_result()
^^^^^^^^^^^^^^^^^^^
File "/home/runner/work/dodal/dodal/.venv/lib/python3.11/site-packages/pluggy/_result.py", line 103, in get_result
raise exc.with_traceback(tb)
File "/home/runner/work/dodal/dodal/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 38, in run_old_style_hookwrapper
res = yield
^^^^^
File "/home/runner/work/dodal/dodal/.venv/lib/python3.11/site-packages/pluggy/_callers.py", line 121, in _multicall
res = hook_impl.function(*args)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/runner/work/dodal/dodal/.venv/lib/python3.11/site-packages/_pytest/python.py", line 166, in pytest_pyfunc_call
result = testfunction(**testargs)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/runner/work/dodal/dodal/.venv/lib/python3.11/site-packages/pytest_asyncio/plugin.py", line 716, in inner
runner.run(coro, context=context)
File "/home/runner/work/_temp/uv-python-dir/cpython-3.11.16-linux-x86_64-gnu/lib/python3.11/asyncio/runners.py", line 118, in run
return self._loop.run_until_complete(task)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/runner/work/_temp/uv-python-dir/cpython-3.11.16-linux-x86_64-gnu/lib/python3.11/asyncio/base_events.py", line 641, in run_until_complete
self.run_forever()
File "/home/runner/work/_temp/uv-python-dir/cpython-3.11.16-linux-x86_64-gnu/lib/python3.11/asyncio/base_events.py", line 608, in run_forever
self._run_once()
File "/home/runner/work/_temp/uv-python-dir/cpython-3.11.16-linux-x86_64-gnu/lib/python3.11/asyncio/base_events.py", line 1898, in _run_once
event_list = self._selector.select(timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/runner/work/_temp/uv-python-dir/cpython-3.11.16-linux-x86_64-gnu/lib/python3.11/selectors.py", line 468, in select
fd_event_list = self._selector.poll(timeout, max_ev)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/runner/work/dodal/dodal/.venv/lib/python3.11/site-packages/pytest_timeout.py", line 317, in handler
timeout_sigalrm(item, settings)
File "/home/runner/work/dodal/dodal/.venv/lib/python3.11/site-packages/pytest_timeout.py", line 502, in timeout_sigalrm
pytest.fail(PYTEST_FAILURE_MESSAGE % settings.timeout)
File "/home/runner/work/dodal/dodal/.venv/lib/python3.11/site-packages/_pytest/outcomes.py", line 163, in __call__
raise Failed(msg=reason, pytrace=pytrace)
Failed: Timeout (>1.0s) from pytest-timeout.
----------------------------- Captured stdout call -----------------------------
+++++++++++++++++++++++++++++++++++ Timeout ++++++++++++++++++++++++++++++++++++
~~~~~~~~~~~~ Stack of Thread-234 (_run_callbacks) (140228931090112) ~~~~~~~~~~~~
File "/home/runner/work/_temp/uv-python-dir/cpython-3.11.16-linux-x86_64-gnu/lib/python3.11/threading.py", line 1002, in _bootstrap
self._bootstrap_inner()
File "/home/runner/work/_temp/uv-python-dir/cpython-3.11.16-linux-x86_64-gnu/lib/python3.11/threading.py", line 1045, in _bootstrap_inner
self.run()
File "/home/runner/work/_temp/uv-python-dir/cpython-3.11.16-linux-x86_64-gnu/lib/python3.11/threading.py", line 982, in run
self._target(*self._args, **self._kwargs)
File "/home/runner/work/dodal/dodal/.venv/lib/python3.11/site-packages/ophyd/status.py", line 247, in _run_callbacks
if not self._settled_event.wait(timeout):
File "/home/runner/work/_temp/uv-python-dir/cpython-3.11.16-linux-x86_64-gnu/lib/python3.11/threading.py", line 629, in wait
signaled = self._cond.wait(timeout)
File "/home/runner/work/_temp/uv-python-dir/cpython-3.11.16-linux-x86_64-gnu/lib/python3.11/threading.py", line 331, in wait
gotit = waiter.acquire(True, timeout)
~~~~~~~~~~~~ Stack of Thread-232 (_run_callbacks) (140229189035712) ~~~~~~~~~~~~
File "/home/runner/work/_temp/uv-python-dir/cpython-3.11.16-linux-x86_64-gnu/lib/python3.11/threading.py", line 1002, in _bootstrap
self._bootstrap_inner()
File "/home/runner/work/_temp/uv-python-dir/cpython-3.11.16-linux-x86_64-gnu/lib/python3.11/threading.py", line 1045, in _bootstrap_inner
self.run()
File "/home/runner/work/_temp/uv-python-dir/cpython-3.11.16-linux-x86_64-gnu/lib/python3.11/threading.py", line 982, in run
self._target(*self._args, **self._kwargs)
File "/home/runner/work/dodal/dodal/.venv/lib/python3.11/site-packages/ophyd/status.py", line 247, in _run_callbacks
if not self._settled_event.wait(timeout):
File "/home/runner/work/_temp/uv-python-dir/cpython-3.11.16-linux-x86_64-gnu/lib/python3.11/threading.py", line 629, in wait
signaled = self._cond.wait(timeout)
File "/home/runner/work/_temp/uv-python-dir/cpython-3.11.16-linux-x86_64-gnu/lib/python3.11/threading.py", line 331, in wait
gotit = waiter.acquire(True, timeout)
~~~~~~~~~~~~ Stack of Thread-229 (_run_callbacks) (140229172254400) ~~~~~~~~~~~~
File "/home/runner/work/_temp/uv-python-dir/cpython-3.11.16-linux-x86_64-gnu/lib/python3.11/threading.py", line 1002, in _bootstrap
self._bootstrap_inner()
File "/home/runner/work/_temp/uv-python-dir/cpython-3.11.16-linux-x86_64-gnu/lib/python3.11/threading.py", line 1045, in _bootstrap_inner
self.run()
File "/home/runner/work/_temp/uv-python-dir/cpython-3.11.16-linux-x86_64-gnu/lib/python3.11/threading.py", line 982, in run
self._target(*self._args, **self._kwargs)
File "/home/runner/work/dodal/dodal/.venv/lib/python3.11/site-packages/ophyd/status.py", line 247, in _run_callbacks
if not self._settled_event.wait(timeout):
File "/home/runner/work/_temp/uv-python-dir/cpython-3.11.16-linux-x86_64-gnu/lib/python3.11/threading.py", line 629, in wait
signaled = self._cond.wait(timeout)
File "/home/runner/work/_temp/uv-python-dir/cpython-3.11.16-linux-x86_64-gnu/lib/python3.11/threading.py", line 331, in wait
gotit = waiter.acquire(True, timeout)
~~~~~~~~~~~~ Stack of Thread-225 (_run_callbacks) (140228914308800) ~~~~~~~~~~~~
File "/home/runner/work/_temp/uv-python-dir/cpython-3.11.16-linux-x86_64-gnu/lib/python3.11/threading.py", line 1002, in _bootstrap
self._bootstrap_inner()
File "/home/runner/work/_temp/uv-python-dir/cpython-3.11.16-linux-x86_64-gnu/lib/python3.11/threading.py", line 1045, in _bootstrap_inner
self.run()
File "/home/runner/work/_temp/uv-python-dir/cpython-3.11.16-linux-x86_64-gnu/lib/python3.11/threading.py", line 982, in run
self._target(*self._args, **self._kwargs)
File "/home/runner/work/dodal/dodal/.venv/lib/python3.11/site-packages/ophyd/status.py", line 247, in _run_callbacks
if not self._settled_event.wait(timeout):
File "/home/runner/work/_temp/uv-python-dir/cpython-3.11.16-linux-x86_64-gnu/lib/python3.11/threading.py", line 629, in wait
signaled = self._cond.wait(timeout)
File "/home/runner/work/_temp/uv-python-dir/cpython-3.11.16-linux-x86_64-gnu/lib/python3.11/threading.py", line 331, in wait
gotit = waiter.acquire(True, timeout)
~~~~~~~~~~~~~~~~ Stack of bluesky-run-engine (140229435455168) ~~~~~~~~~~~~~~~~~
File "/home/runner/work/_temp/uv-python-dir/cpython-3.11.16-linux-x86_64-gnu/lib/python3.11/threading.py", line 1002, in _bootstrap
self._bootstrap_inner()
File "/home/runner/work/_temp/uv-python-dir/cpython-3.11.16-linux-x86_64-gnu/lib/python3.11/threading.py", line 1045, in _bootstrap_inner
self.run()
File "/home/runner/work/_temp/uv-python-dir/cpython-3.11.16-linux-x86_64-gnu/lib/python3.11/threading.py", line 982, in run
self._target(*self._args, **self._kwargs)
File "/home/runner/work/_temp/uv-python-dir/cpython-3.11.16-linux-x86_64-gnu/lib/python3.11/asyncio/base_events.py", line 608, in run_forever
self._run_once()
File "/home/runner/work/_temp/uv-python-dir/cpython-3.11.16-linux-x86_64-gnu/lib/python3.11/asyncio/base_events.py", line 1898, in _run_once
event_list = self._selector.select(timeout)
File "/home/runner/work/_temp/uv-python-dir/cpython-3.11.16-linux-x86_64-gnu/lib/python3.11/selectors.py", line 468, in select
fd_event_list = self._selector.poll(timeout, max_ev)
~~~~~~~~~~~~~~~~ Stack of bluesky-run-engine (140230212449984) ~~~~~~~~~~~~~~~~~
File "/home/runner/work/_temp/uv-python-dir/cpython-3.11.16-linux-x86_64-gnu/lib/python3.11/threading.py", line 1002, in _bootstrap
self._bootstrap_inner()
File "/home/runner/work/_temp/uv-python-dir/cpython-3.11.16-linux-x86_64-gnu/lib/python3.11/threading.py", line 1045, in _bootstrap_inner
self.run()
File "/home/runner/work/_temp/uv-python-dir/cpython-3.11.16-linux-x86_64-gnu/lib/python3.11/threading.py", line 982, in run
self._target(*self._args, **self._kwargs)
File "/home/runner/work/_temp/uv-python-dir/cpython-3.11.16-linux-x86_64-gnu/lib/python3.11/asyncio/base_events.py", line 608, in run_forever
self._run_once()
File "/home/runner/work/_temp/uv-python-dir/cpython-3.11.16-linux-x86_64-gnu/lib/python3.11/asyncio/base_events.py", line 1898, in _run_once
event_list = self._selector.select(timeout)
File "/home/runner/work/_temp/uv-python-dir/cpython-3.11.16-linux-x86_64-gnu/lib/python3.11/selectors.py", line 468, in select
fd_event_list = self._selector.poll(timeout, max_ev)
~~~~~~~~~~~~~~~~~~~ Stack of debug_monitor (140230447322816) ~~~~~~~~~~~~~~~~~~~
File "/home/runner/work/_temp/uv-python-dir/cpython-3.11.16-linux-x86_64-gnu/lib/python3.11/threading.py", line 1002, in _bootstrap
self._bootstrap_inner()
File "/home/runner/work/_temp/uv-python-dir/cpython-3.11.16-linux-x86_64-gnu/lib/python3.11/threading.py", line 1045, in _bootstrap_inner
self.run()
File "/home/runner/work/_temp/uv-python-dir/cpython-3.11.16-linux-x86_64-gnu/lib/python3.11/threading.py", line 982, in run
self._target(*self._args, **self._kwargs)
File "/home/runner/work/dodal/dodal/.venv/lib/python3.11/site-packages/ophyd/_dispatch.py", line 164, in _debug_monitor
time.sleep(self.debug_monitor_interval)
~~~~~~~~~~~~~~~~~~~~~~~ Stack of util3 (140230464104128) ~~~~~~~~~~~~~~~~~~~~~~~
File "/home/runner/work/_temp/uv-python-dir/cpython-3.11.16-linux-x86_64-gnu/lib/python3.11/threading.py", line 1002, in _bootstrap
self._bootstrap_inner()
File "/home/runner/work/_temp/uv-python-dir/cpython-3.11.16-linux-x86_64-gnu/lib/python3.11/threading.py", line 1045, in _bootstrap_inner
self.run()
File "/home/runner/work/dodal/dodal/.venv/lib/python3.11/site-packages/ophyd/_dispatch.py", line 46, in run
callback, args, kwargs = self.queue.get(True, self.timeout)
File "/home/runner/work/_temp/uv-python-dir/cpython-3.11.16-linux-x86_64-gnu/lib/python3.11/queue.py", line 180, in get
self.not_empty.wait(remaining)
File "/home/runner/work/_temp/uv-python-dir/cpython-3.11.16-linux-x86_64-gnu/lib/python3.11/threading.py", line 331, in wait
gotit = waiter.acquire(True, timeout)
----------------------------- Captured stderr call -----------------------------
I 260818 09:40:50.998 fast_grid_scan:308] Applying gridscan parameters...
I 260818 09:40:50.998 fast_grid_scan:324] Gridscan parameters applied, waiting for sets to complete...
I 260818 09:40:51.027 fast_grid_scan:329] Sets confirmed, waiting for validity checks to pass...
------------------------------ Captured log call -------------------------------
INFO Dodal:fast_grid_scan.py:308 Applying gridscan parameters...
INFO Dodal:fast_grid_scan.py:324 Gridscan parameters applied, waiting for sets to complete...
INFO Dodal:fast_grid_scan.py:329 Sets confirmed, waiting for validity checks to pass...
--------------------------- Captured stderr teardown ---------------------------
I 260818 09:40:51.533 beamline_utils:181] Clearing global path provider: None
I 260818 09:40:51.534 beamline_utils:204] Clearing global config client: None
---------------------------- Captured log teardown -----------------------------
INFO Dodal:beamline_utils.py:181 Clearing global path provider: None
INFO Dodal:beamline_utils.py:204 Clearing global config client: None
```

## Acceptance Criteria
* The test passes 100% of the time.

Contributor guide

Open the contributing guide

Research direction

Start by locating the test named test_gridscan_prepare_times_out_for_validity_check and reproduce its zebra parameter case. Inspect the gridscan validity-check timeout and the captured ophyd callback activity shown in the failure. Done means the test no longer flakes or times out in repeated runs and CI.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
testing-qa
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.