DiamondLightSource / DiamondLightSource/blueapi

Running BlueAPI with NumTracker not running causes error difficult to diagnose

Open
#1,610 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
13
Forks
13
Avg merge
2d 11h
Merged PRs (30d)
25

Description

When trying to run blueapi locally using the following configuration

```
api:
url: http://0.0.0.0:8000
env:
metadata:
instrument: i06
sources:
- kind: deviceManager
module: dodal.beamlines.i06_1
mock: True
- kind: planFunctions
module: dodal.plans
- kind: planFunctions
module: dodal.plan_stubs.wrapped
stomp:
enabled: true
url: tcp://localhost:61613/
numtracker:
url: http://localhost:8406/graphql
```

uv run blueapi -c /path/to/blueapi_source.yaml serve

And then running this with blueapi python client, when trying to run a plan such as moving a device it runs okay. However, when trying to run a scan, an error occurs that is not obvious what is causing it.

Client side

```python
>>> pl.step_grid_scan([], params=[(scmc.cart.x, [0, 1, 0.1])])
Traceback (most recent call last):
File "", line 1, in
File "/scratch/bluesky_development/blueapi/src/blueapi/client/client.py", line 159, in __call__
raise PlanFailedError(typ, msg)
blueapi.client.client.PlanFailedError: All connection attempts failed
```

Server side
```
2026-08-04 15:43:43,360 INFO blueapi.worker.task_worker Submitting: task_id='ef0c21cc-bdb1-49e4-8851-caad0170ffeb' task=Task(name='step_grid_scan', params={'detectors': [], 'params': [['scmc.cart.x', [0, 1, 0.1]]]}, metadata={'user': 'Unknown', 'instrument_session': 'cm44150-1'}) request_id=None is_complete=False is_pending=True errors=[] outcome=None
2026-08-04 15:43:43,360 INFO blueapi.worker.task_worker Got new task: task_id='ef0c21cc-bdb1-49e4-8851-caad0170ffeb' task=Task(name='step_grid_scan', params={'detectors': [], 'params': [['scmc.cart.x', [0, 1, 0.1]]]}, metadata={'user': 'Unknown', 'instrument_session': 'cm44150-1'}) request_id=None is_complete=False is_pending=True errors=[] outcome=None
2026-08-04 15:43:43,360 INFO blueapi.worker.task Asked to run plan step_grid_scan with {'detectors': [], 'params': [['scmc.cart.x', [0, 1, 0.1]]]} and metadata {'user': 'Unknown', 'instrument_session': 'cm44150-1'} for all runs
2026-08-04 15:43:43,360 INFO bluesky Executing plan
2026-08-04 15:43:43,360 INFO bluesky.RE.state Change state on from 'idle' -> 'running'
2026-08-04 15:43:43,362 INFO blueapi.service.main 127.0.0.1:35674 PUT /worker/task 200
2026-08-04 15:43:43,368 ERROR bluesky Run aborted
Traceback (most recent call last):
File "/scratch/bluesky_development/blueapi/.venv/lib64/python3.11/site-packages/httpx/_transports/default.py", line 101, in map_httpcore_exceptions
yield
File "/scratch/bluesky_development/blueapi/.venv/lib64/python3.11/site-packages/httpx/_transports/default.py", line 394, in handle_async_request
resp = await self._pool.handle_async_request(req)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/scratch/bluesky_development/blueapi/.venv/lib64/python3.11/site-packages/httpcore/_async/connection_pool.py", line 256, in handle_async_request
raise exc from None
File "/scratch/bluesky_development/blueapi/.venv/lib64/python3.11/site-packages/httpcore/_async/connection_pool.py", line 236, in handle_async_request
response = await connection.handle_async_request(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/scratch/bluesky_development/blueapi/.venv/lib64/python3.11/site-packages/httpcore/_async/connection.py", line 101, in handle_async_request
raise exc
File "/scratch/bluesky_development/blueapi/.venv/lib64/python3.11/site-packages/httpcore/_async/connection.py", line 78, in handle_async_request
stream = await self._connect(request)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/scratch/bluesky_development/blueapi/.venv/lib64/python3.11/site-packages/httpcore/_async/connection.py", line 124, in _connect
stream = await self._network_backend.connect_tcp(**kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/scratch/bluesky_development/blueapi/.venv/lib64/python3.11/site-packages/httpcore/_backends/auto.py", line 31, in connect_tcp
return await self._backend.connect_tcp(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/scratch/bluesky_development/blueapi/.venv/lib64/python3.11/site-packages/httpcore/_backends/anyio.py", line 113, in connect_tcp
with map_exceptions(exc_map):
File "/usr/lib64/python3.11/contextlib.py", line 158, in __exit__
self.gen.throw(typ, value, traceback)
File "/scratch/bluesky_development/blueapi/.venv/lib64/python3.11/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
raise to_exc(exc) from exc
httpcore.ConnectError: All connection attempts failed

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/scratch/bluesky_development/blueapi/.venv/lib64/python3.11/site-packages/bluesky/run_engine.py", line 1616, in _run
msg = self._plan_stack[-1].send(resp)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/scratch/bluesky_development/blueapi/.venv/lib64/python3.11/site-packages/dodal/plans/wrapped.py", line 509, in step_grid_scan
yield from bp.list_grid_scan(
File "/scratch/bluesky_development/blueapi/.venv/lib64/python3.11/site-packages/bluesky/plans.py", line 356, in list_grid_scan
return (yield from scan_nd(detectors, full_cycler, per_step=per_step, md=_md))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/scratch/bluesky_development/blueapi/.venv/lib64/python3.11/site-packages/bluesky/plans.py", line 1168, in scan_nd
return (yield from inner_scan_nd())
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/scratch/bluesky_development/blueapi/.venv/lib64/python3.11/site-packages/bluesky/utils/__init__.py", line 1351, in dec_inner
return (yield from plan)
^^^^^^^^^^^^^^^
File "/scratch/bluesky_development/blueapi/.venv/lib64/python3.11/site-packages/bluesky/preprocessors.py", line 1013, in stage_wrapper
return (yield from finalize_wrapper(inner(), unstage_devices()))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/scratch/bluesky_development/blueapi/.venv/lib64/python3.11/site-packages/bluesky/preprocessors.py", line 548, in finalize_wrapper
ret = yield from plan
^^^^^^^^^^^^^^^
File "/scratch/bluesky_development/blueapi/.venv/lib64/python3.11/site-packages/bluesky/preprocessors.py", line 1011, in inner
return (yield from plan)
^^^^^^^^^^^^^^^
File "/scratch/bluesky_development/blueapi/.venv/lib64/python3.11/site-packages/bluesky/utils/__init__.py", line 1351, in dec_inner
return (yield from plan)
^^^^^^^^^^^^^^^
File "/scratch/bluesky_development/blueapi/.venv/lib64/python3.11/site-packages/bluesky/preprocessors.py", line 362, in run_wrapper
rs_uid = yield from open_run(md)
^^^^^^^^^^^^^^^^^^^^^^^
File "/scratch/bluesky_development/blueapi/.venv/lib64/python3.11/site-packages/bluesky/utils/__init__.py", line 2047, in __iter__
return (yield from self._iter)
^^^^^^^^^^^^^^^^^^^^^
File "/scratch/bluesky_development/blueapi/.venv/lib64/python3.11/site-packages/bluesky/plan_stubs.py", line 1366, in open_run
return (yield Msg("open_run", **(md or {})))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/scratch/bluesky_development/blueapi/.venv/lib64/python3.11/site-packages/bluesky/run_engine.py", line 1683, in _run
new_response = await coro(msg)
^^^^^^^^^^^^^^^
File "/scratch/bluesky_development/blueapi/.venv/lib64/python3.11/site-packages/bluesky/run_engine.py", line 1855, in _open_run
self.md["scan_id"] = await maybe_await(self.scan_id_source(self.md))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/scratch/bluesky_development/blueapi/.venv/lib64/python3.11/site-packages/bluesky/utils/__init__.py", line 2029, in maybe_await
return await ret
^^^^^^^^^
File "/scratch/bluesky_development/blueapi/src/blueapi/core/context.py", line 159, in _update_scan_num
scan = await numtracker.create_scan(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/scratch/bluesky_development/blueapi/src/blueapi/utils/numtracker.py", line 98, in create_scan
response = await client.post(
^^^^^^^^^^^^^^^^^^
File "/scratch/bluesky_development/blueapi/.venv/lib64/python3.11/site-packages/httpx/_client.py", line 1859, in post
return await self.request(
^^^^^^^^^^^^^^^^^^^
File "/scratch/bluesky_development/blueapi/.venv/lib64/python3.11/site-packages/httpx/_client.py", line 1540, in request
return await self.send(request, auth=auth, follow_redirects=follow_redirects)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/scratch/bluesky_development/blueapi/.venv/lib64/python3.11/site-packages/httpx/_client.py", line 1629, in send
response = await self._send_handling_auth(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/scratch/bluesky_development/blueapi/.venv/lib64/python3.11/site-packages/httpx/_client.py", line 1657, in _send_handling_auth
response = await self._send_handling_redirects(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/scratch/bluesky_development/blueapi/.venv/lib64/python3.11/site-packages/httpx/_client.py", line 1694, in _send_handling_redirects
response = await self._send_single_request(request)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/scratch/bluesky_development/blueapi/.venv/lib64/python3.11/site-packages/httpx/_client.py", line 1730, in _send_single_request
response = await transport.handle_async_request(request)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/scratch/bluesky_development/blueapi/.venv/lib64/python3.11/site-packages/httpx/_transports/default.py", line 393, in handle_async_request
with map_httpcore_exceptions():
File "/usr/lib64/python3.11/contextlib.py", line 158, in __exit__
self.gen.throw(typ, value, traceback)
File "/scratch/bluesky_development/blueapi/.venv/lib64/python3.11/site-packages/httpx/_transports/default.py", line 118, in map_httpcore_exceptions
raise mapped_exc(message) from exc
httpx.ConnectError: All connection attempts failed

```

@tpoliaw managed to help me fix it by commenting out numtracker configuration and then the scan would work again, but this traceback is not obvious at all what is causing the issue. Is it possible we could make the traceback user friendly mentioning num tracker?

Contributor guide

Open the contributing guide

Research direction

Start with blueapi/utils/numtracker.py, especially create_scan, and follow its use from blueapi/core/context.py in the traceback. Reproduce the scan with NumTracker unavailable and inspect the existing error handling; done means the failure clearly identifies NumTracker and the relevant connection problem while preserving the existing behavior when the service is available.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.