DiamondLightSource / DiamondLightSource/mx-bluesky

Hyperion supervisor exits if cannot connect to blueapi

Open
#1,688 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
4
Forks
5
Avg merge
1d 30m
Merged PRs (30d)
2

Description

During testing of hyperion, the following exception was raised and caused the supervisor to exit when it could not reach blueapi:

```
"message": "[31m[E 260319 14:35:13.560 run_engine:1755] Run aborted\u001b[0m
Traceback (most recent call last):
File \"/dls_sw/i03/software/bluesky/mx-bluesky_1.6.5/mx-bluesky/.venv/lib/python3.11/site-packages/urllib3/connection.py\", line 204, in _new_conn
sock = connection.create_connection(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File \"/dls_sw/i03/software/bluesky/mx-bluesky_1.6.5/mx-bluesky/.venv/lib/python3.11/site-packages/urllib3/util/connection.py\", line 85, in create_connection
raise err
File \"/dls_sw/i03/software/bluesky/mx-bluesky_1.6.5/mx-bluesky/.venv/lib/python3.11/site-packages/urllib3/util/connection.py\", line 73, in create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File \"/dls_sw/i03/software/bluesky/mx-bluesky_1.6.5/mx-bluesky/.venv/lib/python3.11/site-packages/urllib3/connectionpool.py\", line 787, in urlopen
response = self._make_request(
^^^^^^^^^^^^^^^^^^^
File \"/dls_sw/i03/software/bluesky/mx-bluesky_1.6.5/mx-bluesky/.venv/lib/python3.11/site-packages/urllib3/connectionpool.py\", line 493, in _make_request
conn.request(
File \"/dls_sw/i03/software/bluesky/mx-bluesky_1.6.5/mx-bluesky/.venv/lib/python3.11/site-packages/urllib3/connection.py\", line 500, in request
self.endheaders()
File \"/dls_sw/apps/python/miniforge/4.10.0-0/envs/python3.11/lib/python3.11/http/client.py\", line 1278, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File \"/dls_sw/apps/python/miniforge/4.10.0-0/envs/python3.11/lib/python3.11/http/client.py\", line 1038, in _send_output
self.send(msg)
File \"/dls_sw/apps/python/miniforge/4.10.0-0/envs/python3.11/lib/python3.11/http/client.py\", line 976, in send
self.connect()
File \"/dls_sw/i03/software/bluesky/mx-bluesky_1.6.5/mx-bluesky/.venv/lib/python3.11/site-packages/urllib3/connection.py\", line 331, in connect
self.sock = self._new_conn()
^^^^^^^^^^^^^^^^
File \"/dls_sw/i03/software/bluesky/mx-bluesky_1.6.5/mx-bluesky/.venv/lib/python3.11/site-packages/urllib3/connection.py\", line 219, in _new_conn
raise NewConnectionError(
urllib3.exceptions.NewConnectionError: HTTPConnection(host='localhost', port=5005): Failed to establish a new connection: [Errno 111] Connection refused

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File \"/dls_sw/i03/software/bluesky/mx-bluesky_1.6.5/mx-bluesky/.venv/lib/python3.11/site-packages/requests/adapters.py\", line 644, in send
resp = conn.urlopen(
^^^^^^^^^^^^^
File \"/dls_sw/i03/software/bluesky/mx-bluesky_1.6.5/mx-bluesky/.venv/lib/python3.11/site-packages/urllib3/connectionpool.py\", line 841, in urlopen
retries = retries.increment(
^^^^^^^^^^^^^^^^^^
File \"/dls_sw/i03/software/bluesky/mx-bluesky_1.6.5/mx-bluesky/.venv/lib/python3.11/site-packages/urllib3/util/retry.py\", line 535, in increment
raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=5005): Max retries exceeded with url: /tasks (Caused by NewConnectionError(\"HTTPConnection(host='localhost', port=5005): Failed to establish a new connection: [Errno 111] Connection refused\"))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File \"/dls_sw/i03/software/bluesky/mx-bluesky_1.6.5/mx-bluesky/.venv/lib/python3.11/site-packages/blueapi/client/rest.py\", line 259, in _request_and_deserialize
response = self._pool.request(
^^^^^^^^^^^^^^^^^^^
File \"/dls_sw/i03/software/bluesky/mx-bluesky_1.6.5/mx-bluesky/.venv/lib/python3.11/site-packages/requests/sessions.py\", line 589, in request
resp = self.send(prep, **send_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File \"/dls_sw/i03/software/bluesky/mx-bluesky_1.6.5/mx-bluesky/.venv/lib/python3.11/site-packages/requests/sessions.py\", line 703, in send
r = adapter.send(request, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File \"/dls_sw/i03/software/bluesky/mx-bluesky_1.6.5/mx-bluesky/.venv/lib/python3.11/site-packages/requests/adapters.py\", line 677, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=5005): Max retries exceeded with url: /tasks (Caused by NewConnectionError(\"HTTPConnection(host='localhost', port=5005): Failed to establish a new connection: [Errno 111] Connection refused\"))

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File \"/dls_sw/i03/software/bluesky/mx-bluesky_1.6.5/mx-bluesky/.venv/lib/python3.11/site-packages/bluesky/preprocessors.py\", line 622, in contingency_wrapper
ret = yield from plan
^^^^^^^^^^^^^^^
File \"/dls_sw/i03/software/bluesky/mx-bluesky_1.6.5/mx-bluesky/src/mx_bluesky/hyperion/supervisor/_supervisor.py\", line 81, in decode_and_execute
self._run_task_remotely(task_request)
File \"/dls_sw/i03/software/bluesky/mx-bluesky_1.6.5/mx-bluesky/src/mx_bluesky/hyperion/supervisor/_supervisor.py\", line 127, in _run_task_remotely
task_status = self.blueapi_client.run_task(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File \"/dls_sw/i03/software/bluesky/mx-bluesky_1.6.5/mx-bluesky/.venv/lib/python3.11/site-packages/observability_utils/tracing/decorators.py\", line 151, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File \"/dls_sw/i03/software/bluesky/mx-bluesky_1.6.5/mx-bluesky/.venv/lib/python3.11/site-packages/blueapi/client/client.py\", line 468, in run_task
task_response = self._rest.create_task(task)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File \"/dls_sw/i03/software/bluesky/mx-bluesky_1.6.5/mx-bluesky/.venv/lib/python3.11/site-packages/blueapi/client/rest.py\", line 188, in create_task
return self._request_and_deserialize(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File \"/dls_sw/i03/software/bluesky/mx-bluesky_1.6.5/mx-bluesky/.venv/lib/python3.11/site-packages/observability_utils/tracing/decorators.py\", line 151, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File \"/dls_sw/i03/software/bluesky/mx-bluesky_1.6.5/mx-bluesky/.venv/lib/python3.11/site-packages/blueapi/client/rest.py\", line 268, in _request_and_deserialize
raise ServiceUnavailableError() from ce
blueapi.client.rest.ServiceUnavailableError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File \"/dls_sw/i03/software/bluesky/mx-bluesky_1.6.5/mx-bluesky/.venv/lib/python3.11/site-packages/bluesky/preprocessors.py\", line 622, in contingency_wrapper
ret = yield from plan
^^^^^^^^^^^^^^^
File \"/dls_sw/i03/software/bluesky/mx-bluesky_1.6.5/mx-bluesky/src/mx_bluesky/hyperion/baton_handler.py\", line 93, in collect
yield from bpp.contingency_wrapper(
File \"/dls_sw/i03/software/bluesky/mx-bluesky_1.6.5/mx-bluesky/.venv/lib/python3.11/site-packages/bluesky/preprocessors.py\", line 632, in contingency_wrapper
ret = yield from except_plan(e)
^^^^^^^^^^^^^^^^^^^^^^^^^
File \"/dls_sw/i03/software/bluesky/mx-bluesky_1.6.5/mx-bluesky/src/mx_bluesky/hyperion/baton_handler.py\", line 128, in trap_default_state_exception
raise PlanError(\"Unexpected exception from UDC Default State plan\") from e
mx_bluesky.hyperion.plan_runner.PlanError: Unexpected exception from UDC Default State plan

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File \"/dls_sw/i03/software/bluesky/mx-bluesky_1.6.5/mx-bluesky/.venv/lib/python3.11/site-packages/urllib3/connection.py\", line 204, in _new_conn
sock = connection.create_connection(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File \"/dls_sw/i03/software/bluesky/mx-bluesky_1.6.5/mx-bluesky/.venv/lib/python3.11/site-packages/urllib3/util/connection.py\", line 85, in create_connection
raise err
File \"/dls_sw/i03/software/bluesky/mx-bluesky_1.6.5/mx-bluesky/.venv/lib/python3.11/site-packages/urllib3/util/connection.py\", line 73, in create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File \"/dls_sw/i03/software/bluesky/mx-bluesky_1.6.5/mx-bluesky/.venv/lib/python3.11/site-packages/urllib3/connectionpool.py\", line 787, in urlopen
response = self._make_request(
^^^^^^^^^^^^^^^^^^^
File \"/dls_sw/i03/software/bluesky/mx-bluesky_1.6.5/mx-bluesky/.venv/lib/python3.11/site-packages/urllib3/connectionpool.py\", line 493, in _make_request
conn.request(
File \"/dls_sw/i03/software/bluesky/mx-bluesky_1.6.5/mx-bluesky/.venv/lib/python3.11/site-packages/urllib3/connection.py\", line 500, in request
self.endheaders()
File \"/dls_sw/apps/python/miniforge/4.10.0-0/envs/python3.11/lib/python3.11/http/client.py\", line 1278, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File \"/dls_sw/apps/python/miniforge/4.10.0-0/envs/python3.11/lib/python3.11/http/client.py\", line 1038, in _send_output
self.send(msg)
File \"/dls_sw/apps/python/miniforge/4.10.0-0/envs/python3.11/lib/python3.11/http/client.py\", line 976, in send
self.connect()
File \"/dls_sw/i03/software/bluesky/mx-bluesky_1.6.5/mx-bluesky/.venv/lib/python3.11/site-packages/urllib3/connection.py\", line 331, in connect
self.sock = self._new_conn()
^^^^^^^^^^^^^^^^
File \"/dls_sw/i03/software/bluesky/mx-bluesky_1.6.5/mx-bluesky/.venv/lib/python3.11/site-packages/urllib3/connection.py\", line 219, in _new_conn
raise NewConnectionError(
urllib3.exceptions.NewConnectionError: HTTPConnection(host='localhost', port=5005): Failed to establish a new connection: [Errno 111] Connection refused

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File \"/dls_sw/i03/software/bluesky/mx-bluesky_1.6.5/mx-bluesky/.venv/lib/python3.11/site-packages/requests/adapters.py\", line 644, in send
resp = conn.urlopen(
^^^^^^^^^^^^^
File \"/dls_sw/i03/software/bluesky/mx-bluesky_1.6.5/mx-bluesky/.venv/lib/python3.11/site-packages/urllib3/connectionpool.py\", line 841, in urlopen
retries = retries.increment(
^^^^^^^^^^^^^^^^^^
File \"/dls_sw/i03/software/bluesky/mx-bluesky_1.6.5/mx-bluesky/.venv/lib/python3.11/site-packages/urllib3/util/retry.py\", line 535, in increment
raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=5005): Max retries exceeded with url: /tasks (Caused by NewConnectionError(\"HTTPConnection(host='localhost', port=5005): Failed to establish a new connection: [Errno 111] Connection refused\"))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File \"/dls_sw/i03/software/bluesky/mx-bluesky_1.6.5/mx-bluesky/.venv/lib/python3.11/site-packages/blueapi/client/rest.py\", line 259, in _request_and_deserialize
response = self._pool.request(
^^^^^^^^^^^^^^^^^^^
File \"/dls_sw/i03/software/bluesky/mx-bluesky_1.6.5/mx-bluesky/.venv/lib/python3.11/site-packages/requests/sessions.py\", line 589, in request
resp = self.send(prep, **send_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File \"/dls_sw/i03/software/bluesky/mx-bluesky_1.6.5/mx-bluesky/.venv/lib/python3.11/site-packages/requests/sessions.py\", line 703, in send
r = adapter.send(request, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File \"/dls_sw/i03/software/bluesky/mx-bluesky_1.6.5/mx-bluesky/.venv/lib/python3.11/site-packages/requests/adapters.py\", line 677, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=5005): Max retries exceeded with url: /tasks (Caused by NewConnectionError(\"HTTPConnection(host='localhost', port=5005): Failed to establish a new connection: [Errno 111] Connection refused\"))

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File \"/dls_sw/i03/software/bluesky/mx-bluesky_1.6.5/mx-bluesky/.venv/lib/python3.11/site-packages/bluesky/run_engine.py\", line 1610, in _run
msg = self._plan_stack[-1].send(resp)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File \"/dls_sw/i03/software/bluesky/mx-bluesky_1.6.5/mx-bluesky/src/mx_bluesky/hyperion/baton_handler.py\", line 131, in collect_then_release
yield from bpp.contingency_wrapper(
File \"/dls_sw/i03/software/bluesky/mx-bluesky_1.6.5/mx-bluesky/.venv/lib/python3.11/site-packages/bluesky/preprocessors.py\", line 653, in contingency_wrapper
yield from final_plan()
File \"/dls_sw/i03/software/bluesky/mx-bluesky_1.6.5/mx-bluesky/src/mx_bluesky/hyperion/baton_handler.py\", line 115, in clean_up_and_release_baton
yield from runner.decode_and_execute(None, [UDCCleanup()])
File \"/dls_sw/i03/software/bluesky/mx-bluesky_1.6.5/mx-bluesky/src/mx_bluesky/hyperion/supervisor/_supervisor.py\", line 88, in decode_and_execute
self._run_task_remotely(task_request)
File \"/dls_sw/i03/software/bluesky/mx-bluesky_1.6.5/mx-bluesky/src/mx_bluesky/hyperion/supervisor/_supervisor.py\", line 127, in _run_task_remotely
task_status = self.blueapi_client.run_task(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File \"/dls_sw/i03/software/bluesky/mx-bluesky_1.6.5/mx-bluesky/.venv/lib/python3.11/site-packages/observability_utils/tracing/decorators.py\", line 151, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File \"/dls_sw/i03/software/bluesky/mx-bluesky_1.6.5/mx-bluesky/.venv/lib/python3.11/site-packages/blueapi/client/client.py\", line 468, in run_task
task_response = self._rest.create_task(task)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File \"/dls_sw/i03/software/bluesky/mx-bluesky_1.6.5/mx-bluesky/.venv/lib/python3.11/site-packages/blueapi/client/rest.py\", line 188, in create_task
return self._request_and_deserialize(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File \"/dls_sw/i03/software/bluesky/mx-bluesky_1.6.5/mx-bluesky/.venv/lib/python3.11/site-packages/observability_utils/tracing/decorators.py\", line 151, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File \"/dls_sw/i03/software/bluesky/mx-bluesky_1.6.5/mx-bluesky/.venv/lib/python3.11/site-packages/blueapi/client/rest.py\", line 268, in _request_and_deserialize
raise ServiceUnavailableError() from ce
blueapi.client.rest.ServiceUnavailableError",
"gl2_source_input": "66264d79952b2458ee4b4bc4",
"gl2_processing_timestamp": "2026-03-19 14:35:13.568",
"file": "/dls_sw/i03/software/bluesky/mx-bluesky_1.6.5/mx-bluesky/.venv/lib/python3.11/site-packages/bluesky/run_engine.py",
"RE": "",
"full_message": "Traceback (most recent call last):
File \"/dls_sw/i03/software/bluesky/mx-bluesky_1.6.5/mx-bluesky/.venv/lib/python3.11/site-packages/urllib3/connection.py\", line 204, in _new_conn
sock = connection.create_connection(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File \"/dls_sw/i03/software/bluesky/mx-bluesky_1.6.5/mx-bluesky/.venv/lib/python3.11/site-packages/urllib3/util/connection.py\", line 85, in create_connection
raise err
File \"/dls_sw/i03/software/bluesky/mx-bluesky_1.6.5/mx-bluesky/.venv/lib/python3.11/site-packages/urllib3/util/connection.py\", line 73, in create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File \"/dls_sw/i03/software/bluesky/mx-bluesky_1.6.5/mx-bluesky/.venv/lib/python3.11/site-packages/urllib3/connectionpool.py\", line 787, in urlopen
response = self._make_request(
^^^^^^^^^^^^^^^^^^^
File \"/dls_sw/i03/software/bluesky/mx-bluesky_1.6.5/mx-bluesky/.venv/lib/python3.11/site-packages/urllib3/connectionpool.py\", line 493, in _make_request
conn.request(
File \"/dls_sw/i03/software/bluesky/mx-bluesky_1.6.5/mx-bluesky/.venv/lib/python3.11/site-packages/urllib3/connection.py\", line 500, in request
self.endheaders()
File \"/dls_sw/apps/python/miniforge/4.10.0-0/envs/python3.11/lib/python3.11/http/client.py\", line 1278, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File \"/dls_sw/apps/python/miniforge/4.10.0-0/envs/python3.11/lib/python3.11/http/client.py\", line 1038, in _send_output
self.send(msg)
File \"/dls_sw/apps/python/miniforge/4.10.0-0/envs/python3.11/lib/python3.11/http/client.py\", line 976, in send
self.connect()
File \"/dls_sw/i03/software/bluesky/mx-bluesky_1.6.5/mx-bluesky/.venv/lib/python3.11/site-packages/urllib3/connection.py\", line 331, in connect
self.sock = self._new_conn()
^^^^^^^^^^^^^^^^
File \"/dls_sw/i03/software/bluesky/mx-bluesky_1.6.5/mx-bluesky/.venv/lib/python3.11/site-packages/urllib3/connection.py\", line 219, in _new_conn
raise NewConnectionError(
urllib3.exceptions.NewConnectionError: HTTPConnection(host='localhost', port=5005): Failed to establish a new connection: [Errno 111] Connection refused

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File \"/dls_sw/i03/software/bluesky/mx-bluesky_1.6.5/mx-bluesky/.venv/lib/python3.11/site-packages/requests/adapters.py\", line 644, in send
resp = conn.urlopen(
^^^^^^^^^^^^^
File \"/dls_sw/i03/software/bluesky/mx-bluesky_1.6.5/mx-bluesky/.venv/lib/python3.11/site-packages/urllib3/connectionpool.py\", line 841, in urlopen
retries = retries.increment(
^^^^^^^^^^^^^^^^^^
File \"/dls_sw/i03/software/bluesky/mx-bluesky_1.6.5/mx-bluesky/.venv/lib/python3.11/site-packages/urllib3/util/retry.py\", line 535, in increment
raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=5005): Max retries exceeded with url: /tasks (Caused by NewConnectionError(\"HTTPConnection(host='localhost', port=5005): Failed to establish a new connection: [Errno 111] Connection refused\"))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File \"/dls_sw/i03/software/bluesky/mx-bluesky_1.6.5/mx-bluesky/.venv/lib/python3.11/site-packages/blueapi/client/rest.py\", line 259, in _request_and_deserialize
response = self._pool.request(
^^^^^^^^^^^^^^^^^^^
File \"/dls_sw/i03/software/bluesky/mx-bluesky_1.6.5/mx-bluesky/.venv/lib/python3.11/site-packages/requests/sessions.py\", line 589, in request
resp = self.send(prep, **send_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File \"/dls_sw/i03/software/bluesky/mx-bluesky_1.6.5/mx-bluesky/.venv/lib/python3.11/site-packages/requests/sessions.py\", line 703, in send
r = adapter.send(request, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File \"/dls_sw/i03/software/bluesky/mx-bluesky_1.6.5/mx-bluesky/.venv/lib/python3.11/site-packages/requests/adapters.py\", line 677, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=5005): Max retries exceeded with url: /tasks (Caused by NewConnectionError(\"HTTPConnection(host='localhost', port=5005): Failed to establish a new connection: [Errno 111] Connection refused\"))

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File \"/dls_sw/i03/software/bluesky/mx-bluesky_1.6.5/mx-bluesky/.venv/lib/python3.11/site-packages/bluesky/preprocessors.py\", line 622, in contingency_wrapper
ret = yield from plan
^^^^^^^^^^^^^^^
File \"/dls_sw/i03/software/bluesky/mx-bluesky_1.6.5/mx-bluesky/src/mx_bluesky/hyperion/supervisor/_supervisor.py\", line 81, in decode_and_execute
self._run_task_remotely(task_request)
File \"/dls_sw/i03/software/bluesky/mx-bluesky_1.6.5/mx-bluesky/src/mx_bluesky/hyperion/supervisor/_supervisor.py\", line 127, in _run_task_remotely
task_status = self.blueapi_client.run_task(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File \"/dls_sw/i03/software/bluesky/mx-bluesky_1.6.5/mx-bluesky/.venv/lib/python3.11/site-packages/observability_utils/tracing/decorators.py\", line 151, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File \"/dls_sw/i03/software/bluesky/mx-bluesky_1.6.5/mx-bluesky/.venv/lib/python3.11/site-packages/blueapi/client/client.py\", line 468, in run_task
task_response = self._rest.create_task(task)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File \"/dls_sw/i03/software/bluesky/mx-bluesky_1.6.5/mx-bluesky/.venv/lib/python3.11/site-packages/blueapi/client/rest.py\", line 188, in create_task
return self._request_and_deserialize(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File \"/dls_sw/i03/software/bluesky/mx-bluesky_1.6.5/mx-bluesky/.venv/lib/python3.11/site-packages/observability_utils/tracing/decorators.py\", line 151, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File \"/dls_sw/i03/software/bluesky/mx-bluesky_1.6.5/mx-bluesky/.venv/lib/python3.11/site-packages/blueapi/client/rest.py\", line 268, in _request_and_deserialize
raise ServiceUnavailableError() from ce
blueapi.client.rest.ServiceUnavailableError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File \"/dls_sw/i03/software/bluesky/mx-bluesky_1.6.5/mx-bluesky/.venv/lib/python3.11/site-packages/bluesky/preprocessors.py\", line 622, in contingency_wrapper
ret = yield from plan
^^^^^^^^^^^^^^^
File \"/dls_sw/i03/software/bluesky/mx-bluesky_1.6.5/mx-bluesky/src/mx_bluesky/hyperion/baton_handler.py\", line 93, in collect
yield from bpp.contingency_wrapper(
File \"/dls_sw/i03/software/bluesky/mx-bluesky_1.6.5/mx-bluesky/.venv/lib/python3.11/site-packages/bluesky/preprocessors.py\", line 632, in contingency_wrapper
ret = yield from except_plan(e)
^^^^^^^^^^^^^^^^^^^^^^^^^
File \"/dls_sw/i03/software/bluesky/mx-bluesky_1.6.5/mx-bluesky/src/mx_bluesky/hyperion/baton_handler.py\", line 128, in trap_default_state_exception
raise PlanError(\"Unexpected exception from UDC Default State plan\") from e
mx_bluesky.hyperion.plan_runner.PlanError: Unexpected exception from UDC Default State plan

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File \"/dls_sw/i03/software/bluesky/mx-bluesky_1.6.5/mx-bluesky/.venv/lib/python3.11/site-packages/urllib3/connection.py\", line 204, in _new_conn
sock = connection.create_connection(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File \"/dls_sw/i03/software/bluesky/mx-bluesky_1.6.5/mx-bluesky/.venv/lib/python3.11/site-packages/urllib3/util/connection.py\", line 85, in create_connection
raise err
File \"/dls_sw/i03/software/bluesky/mx-bluesky_1.6.5/mx-bluesky/.venv/lib/python3.11/site-packages/urllib3/util/connection.py\", line 73, in create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File \"/dls_sw/i03/software/bluesky/mx-bluesky_1.6.5/mx-bluesky/.venv/lib/python3.11/site-packages/urllib3/connectionpool.py\", line 787, in urlopen
response = self._make_request(
^^^^^^^^^^^^^^^^^^^
File \"/dls_sw/i03/software/bluesky/mx-bluesky_1.6.5/mx-bluesky/.venv/lib/python3.11/site-packages/urllib3/connectionpool.py\", line 493, in _make_request
conn.request(
File \"/dls_sw/i03/software/bluesky/mx-bluesky_1.6.5/mx-bluesky/.venv/lib/python3.11/site-packages/urllib3/connection.py\", line 500, in request
self.endheaders()
File \"/dls_sw/apps/python/miniforge/4.10.0-0/envs/python3.11/lib/python3.11/http/client.py\", line 1278, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File \"/dls_sw/apps/python/miniforge/4.10.0-0/envs/python3.11/lib/python3.11/http/client.py\", line 1038, in _send_output
self.send(msg)
File \"/dls_sw/apps/python/miniforge/4.10.0-0/envs/python3.11/lib/python3.11/http/client.py\", line 976, in send
self.connect()
File \"/dls_sw/i03/software/bluesky/mx-bluesky_1.6.5/mx-bluesky/.venv/lib/python3.11/site-packages/urllib3/connection.py\", line 331, in connect
self.sock = self._new_conn()
^^^^^^^^^^^^^^^^
File \"/dls_sw/i03/software/bluesky/mx-bluesky_1.6.5/mx-bluesky/.venv/lib/python3.11/site-packages/urllib3/connection.py\", line 219, in _new_conn
raise NewConnectionError(
urllib3.exceptions.NewConnectionError: HTTPConnection(host='localhost', port=5005): Failed to establish a new connection: [Errno 111] Connection refused

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File \"/dls_sw/i03/software/bluesky/mx-bluesky_1.6.5/mx-bluesky/.venv/lib/python3.11/site-packages/requests/adapters.py\", line 644, in send
resp = conn.urlopen(
^^^^^^^^^^^^^
File \"/dls_sw/i03/software/bluesky/mx-bluesky_1.6.5/mx-bluesky/.venv/lib/python3.11/site-packages/urllib3/connectionpool.py\", line 841, in urlopen
retries = retries.increment(
^^^^^^^^^^^^^^^^^^
File \"/dls_sw/i03/software/bluesky/mx-bluesky_1.6.5/mx-bluesky/.venv/lib/python3.11/site-packages/urllib3/util/retry.py\", line 535, in increment
raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=5005): Max retries exceeded with url: /tasks (Caused by NewConnectionError(\"HTTPConnection(host='localhost', port=5005): Failed to establish a new connection: [Errno 111] Connection refused\"))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File \"/dls_sw/i03/software/bluesky/mx-bluesky_1.6.5/mx-bluesky/.venv/lib/python3.11/site-packages/blueapi/client/rest.py\", line 259, in _request_and_deserialize
response = self._pool.request(
^^^^^^^^^^^^^^^^^^^
File \"/dls_sw/i03/software/bluesky/mx-bluesky_1.6.5/mx-bluesky/.venv/lib/python3.11/site-packages/requests/sessions.py\", line 589, in request
resp = self.send(prep, **send_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File \"/dls_sw/i03/software/bluesky/mx-bluesky_1.6.5/mx-bluesky/.venv/lib/python3.11/site-packages/requests/sessions.py\", line 703, in send
r = adapter.send(request, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File \"/dls_sw/i03/software/bluesky/mx-bluesky_1.6.5/mx-bluesky/.venv/lib/python3.11/site-packages/requests/adapters.py\", line 677, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=5005): Max retries exceeded with url: /tasks (Caused by NewConnectionError(\"HTTPConnection(host='localhost', port=5005): Failed to establish a new connection: [Errno 111] Connection refused\"))

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File \"/dls_sw/i03/software/bluesky/mx-bluesky_1.6.5/mx-bluesky/.venv/lib/python3.11/site-packages/bluesky/run_engine.py\", line 1610, in _run
msg = self._plan_stack[-1].send(resp)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File \"/dls_sw/i03/software/bluesky/mx-bluesky_1.6.5/mx-bluesky/src/mx_bluesky/hyperion/baton_handler.py\", line 131, in collect_then_release
yield from bpp.contingency_wrapper(
File \"/dls_sw/i03/software/bluesky/mx-bluesky_1.6.5/mx-bluesky/.venv/lib/python3.11/site-packages/bluesky/preprocessors.py\", line 653, in contingency_wrapper
yield from final_plan()
File \"/dls_sw/i03/software/bluesky/mx-bluesky_1.6.5/mx-bluesky/src/mx_bluesky/hyperion/baton_handler.py\", line 115, in clean_up_and_release_baton
yield from runner.decode_and_execute(None, [UDCCleanup()])
File \"/dls_sw/i03/software/bluesky/mx-bluesky_1.6.5/mx-bluesky/src/mx_bluesky/hyperion/supervisor/_supervisor.py\", line 88, in decode_and_execute
self._run_task_remotely(task_request)
File \"/dls_sw/i03/software/bluesky/mx-bluesky_1.6.5/mx-bluesky/src/mx_bluesky/hyperion/supervisor/_supervisor.py\", line 127, in _run_task_remotely
task_status = self.blueapi_client.run_task(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File \"/dls_sw/i03/software/bluesky/mx-bluesky_1.6.5/mx-bluesky/.venv/lib/python3.11/site-packages/observability_utils/tracing/decorators.py\", line 151, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File \"/dls_sw/i03/software/bluesky/mx-bluesky_1.6.5/mx-bluesky/.venv/lib/python3.11/site-packages/blueapi/client/client.py\", line 468, in run_task
task_response = self._rest.create_task(task)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File \"/dls_sw/i03/software/bluesky/mx-bluesky_1.6.5/mx-bluesky/.venv/lib/python3.11/site-packages/blueapi/client/rest.py\", line 188, in create_task
return self._request_and_deserialize(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File \"/dls_sw/i03/software/bluesky/mx-bluesky_1.6.5/mx-bluesky/.venv/lib/python3.11/site-packages/observability_utils/tracing/decorators.py\", line 151, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File \"/dls_sw/i03/software/bluesky/mx-bluesky_1.6.5/mx-bluesky/.venv/lib/python3.11/site-packages/blueapi/client/rest.py\", line 268, in _request_and_deserialize
raise ServiceUnavailableError() from ce
blueapi.client.rest.ServiceUnavailableError",
```

## Acceptance Criteria
- Hyperion supervisor does not exit if it cannot connect to blueapi, instead if it cannot connect to blueapi, UDC should stop and hyperion supervisor should hand back the baton and await the baton being returned.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.