BYOS: KeyError in Galaxy object store due to S3 connection timeout and endpoint is missing or dead

**Describe the bug**
I had configured a user object store, and for testing, I had some test runs. After a few days, the endpoint kind of died or ran out of privileges, and I did not notice that. When I tried to export a history whose datasets are stored in the user object store, which is no longer reachable or accessible, the UI does not show that there is a problem with the storage endpoint; it just processes it for a while and fails with no additional info. I found the traceback (see below) from Celery complaining about a timeout and then realized that there is an issue with my storage endpoint. It would be nice if we could somehow show some info on the UI to the user that it's not Galaxy but the user-defined object store that has the issues.

_I also remember seeing a few more errors like these in different scenarios and I will try to reproduce them and create additional issues_

**Galaxy Version and/or server at which you observed the bug**
Galaxy version: 24.2

**To Reproduce**
Steps to reproduce the behavior:
1. Configure a User object store via your user preferences
2. Select the newly added object store as your preferred storage at the history level
3. Run some jobs while all jobs uses the newly added object store
4. Disconnect the object store (shutdown the object store endpoint or the bucket or something; simulate it)
5. Export this history
6. The export history does not show what the actual problem is and it just keeps processing and after a while it fails with no clear description.

**Expected behavior**
The UI shows that the storage endpoint where the history datasets are located is timed out, failing, or something similar. It would be nice to see where the history is exported to in the `Export Records` section (I think we have this info somewhere in the DB), like the object storage ID or Name, if possible.

**Additional context**

Traceback:

```
Apr 09 10:48:32 celery-0.bi.privat watchmedo[3485]: [2025-04-09 10:48:32,159: INFO/MainProcess] Scaling down 6 processes.
Apr 09 10:48:37 celery-0.bi.privat watchmedo[6450]: [2025-04-09 10:48:37,527: DEBUG/main] Exception received when sending HTTP request.
Apr 09 10:48:37 celery-0.bi.privat watchmedo[6450]: Traceback (most recent call last):
Apr 09 10:48:37 celery-0.bi.privat watchmedo[6450]: File "/opt/galaxy/server/lib/galaxy/objectstore/__init__.py", line 1543, in _resolve_backend
Apr 09 10:48:37 celery-0.bi.privat watchmedo[6450]: return self.backends[object_store_id]
Apr 09 10:48:37 celery-0.bi.privat watchmedo[6450]: ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
Apr 09 10:48:37 celery-0.bi.privat watchmedo[6450]: KeyError: 'user_objects://f458b1d7-9330-488b-988a-8289e516a208'
Apr 09 10:48:37 celery-0.bi.privat watchmedo[6450]: During handling of the above exception, another exception occurred:
Apr 09 10:48:37 celery-0.bi.privat watchmedo[6450]: Traceback (most recent call last):
Apr 09 10:48:37 celery-0.bi.privat watchmedo[6450]: File "/opt/galaxy/venv/lib/python3.11/site-packages/urllib3/connection.py", line 174, in _new_conn
Apr 09 10:48:37 celery-0.bi.privat watchmedo[6450]: conn = connection.create_connection(
Apr 09 10:48:37 celery-0.bi.privat watchmedo[6450]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Apr 09 10:48:37 celery-0.bi.privat watchmedo[6450]: File "/opt/galaxy/venv/lib/python3.11/site-packages/urllib3/util/connection.py", line 95, in create_connection
Apr 09 10:48:37 celery-0.bi.privat watchmedo[6450]: raise err
Apr 09 10:48:37 celery-0.bi.privat watchmedo[6450]: File "/opt/galaxy/venv/lib/python3.11/site-packages/urllib3/util/connection.py", line 85, in create_connection
Apr 09 10:48:37 celery-0.bi.privat watchmedo[6450]: sock.connect(sa)
Apr 09 10:48:37 celery-0.bi.privat watchmedo[6450]: TimeoutError: timed out
Apr 09 10:48:37 celery-0.bi.privat watchmedo[6450]: During handling of the above exception, another exception occurred:
Apr 09 10:48:37 celery-0.bi.privat watchmedo[6450]: Traceback (most recent call last):
Apr 09 10:48:37 celery-0.bi.privat watchmedo[6450]: File "/opt/galaxy/venv/lib/python3.11/site-packages/botocore/httpsession.py", line 464, in send
Apr 09 10:48:37 celery-0.bi.privat watchmedo[6450]: urllib_response = conn.urlopen(
Apr 09 10:48:37 celery-0.bi.privat watchmedo[6450]: ^^^^^^^^^^^^^
Apr 09 10:48:37 celery-0.bi.privat watchmedo[6450]: File "/opt/galaxy/venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 802, in urlopen
Apr 09 10:48:37 celery-0.bi.privat watchmedo[6450]: retries = retries.increment(
Apr 09 10:48:37 celery-0.bi.privat watchmedo[6450]: ^^^^^^^^^^^^^^^^^^
Apr 09 10:48:37 celery-0.bi.privat watchmedo[6450]: File "/opt/galaxy/venv/lib/python3.11/site-packages/urllib3/util/retry.py", line 527, in increment
Apr 09 10:48:37 celery-0.bi.privat watchmedo[6450]: raise six.reraise(type(error), error, _stacktrace)
Apr 09 10:48:37 celery-0.bi.privat watchmedo[6450]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Apr 09 10:48:37 celery-0.bi.privat watchmedo[6450]: File "/opt/galaxy/venv/lib/python3.11/site-packages/urllib3/packages/six.py", line 770, in reraise
Apr 09 10:48:37 celery-0.bi.privat watchmedo[6450]: raise value
Apr 09 10:48:37 celery-0.bi.privat watchmedo[6450]: File "/opt/galaxy/venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 716, in urlopen
Apr 09 10:48:37 celery-0.bi.privat watchmedo[6450]: httplib_response = self._make_request(
Apr 09 10:48:37 celery-0.bi.privat watchmedo[6450]: ^^^^^^^^^^^^^^^^^^^
Apr 09 10:48:37 celery-0.bi.privat watchmedo[6450]: File "/opt/galaxy/venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 404, in _make_request
Apr 09 10:48:37 celery-0.bi.privat watchmedo[6450]: self._validate_conn(conn)
Apr 09 10:48:37 celery-0.bi.privat watchmedo[6450]: File "/opt/galaxy/venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 1061, in _validate_conn
Apr 09 10:48:37 celery-0.bi.privat watchmedo[6450]: conn.connect()
Apr 09 10:48:37 celery-0.bi.privat watchmedo[6450]: File "/opt/galaxy/venv/lib/python3.11/site-packages/urllib3/connection.py", line 363, in connect
Apr 09 10:48:37 celery-0.bi.privat watchmedo[6450]: self.sock = conn = self._new_conn()
Apr 09 10:48:37 celery-0.bi.privat watchmedo[6450]: ^^^^^^^^^^^^^^^^
Apr 09 10:48:37 celery-0.bi.privat watchmedo[6450]: File "/opt/galaxy/venv/lib/python3.11/site-packages/urllib3/connection.py", line 179, in _new_conn
Apr 09 10:48:37 celery-0.bi.privat watchmedo[6450]: raise ConnectTimeoutError(
Apr 09 10:48:37 celery-0.bi.privat watchmedo[6450]: urllib3.exceptions.ConnectTimeoutError: (, 'Connection to usegalaxy-esg-api.test.fedcloud.eu timed out. (connect timeout=60)')
Apr 09 10:48:37 celery-0.bi.privat watchmedo[6450]: During handling of the above exception, another exception occurred:
Apr 09 10:48:37 celery-0.bi.privat watchmedo[6450]: Traceback (most recent call last):
Apr 09 10:48:37 celery-0.bi.privat watchmedo[6450]: File "/opt/galaxy/venv/lib/python3.11/site-packages/botocore/endpoint.py", line 279, in _do_get_response
Apr 09 10:48:37 celery-0.bi.privat watchmedo[6450]: http_response = self._send(request)
Apr 09 10:48:37 celery-0.bi.privat watchmedo[6450]: ^^^^^^^^^^^^^^^^^^^
Apr 09 10:48:37 celery-0.bi.privat watchmedo[6450]: File "/opt/galaxy/venv/lib/python3.11/site-packages/botocore/endpoint.py", line 383, in _send
Apr 09 10:48:37 celery-0.bi.privat watchmedo[6450]: return self.http_session.send(request)
Apr 09 10:48:37 celery-0.bi.privat watchmedo[6450]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Apr 09 10:48:37 celery-0.bi.privat watchmedo[6450]: File "/opt/galaxy/venv/lib/python3.11/site-packages/botocore/httpsession.py", line 499, in send
Apr 09 10:48:37 celery-0.bi.privat watchmedo[6450]: raise ConnectTimeoutError(endpoint_url=request.url, error=e)
Apr 09 10:48:37 celery-0.bi.privat watchmedo[6450]: botocore.exceptions.ConnectTimeoutError: Connect timeout on endpoint URL: "https://usegalaxy-esg-api.test.fedcloud.eu:443/usegalaxy-esg-byos-minio-test"
Apr 09 10:48:37 celery-0.bi.privat watchmedo[6450]: [2025-04-09 10:48:37,547: DEBUG/main] Event needs-retry.s3.HeadBucket: calling handler
Apr 09 10:48:37 celery-0.bi.privat watchmedo[6450]: [2025-04-09 10:48:37,547: DEBUG/main] retry needed, retryable exception caught: Connect timeout on endpoint URL: "https://usegalaxy-esg-api.test.fedcloud.eu:443/usegalaxy-esg-byos-minio-test"
Apr 09 10:48:37 celery-0.bi.privat watchmedo[6450]: Traceback (most recent call last):
Apr 09 10:48:37 celery-0.bi.privat watchmedo[6450]: File "/opt/galaxy/server/lib/galaxy/objectstore/__init__.py", line 1543, in _resolve_backend
Apr 09 10:48:37 celery-0.bi.privat watchmedo[6450]: return self.backends[object_store_id]
Apr 09 10:48:37 celery-0.bi.privat watchmedo[6450]: ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
Apr 09 10:48:37 celery-0.bi.privat watchmedo[6450]: KeyError: 'user_objects://f458b1d7-9330-488b-988a-8289e516a208'
Apr 09 10:48:37 celery-0.bi.privat watchmedo[6450]: During handling of the above exception, another exception occurred:
Apr 09 10:48:37 celery-0.bi.privat watchmedo[6450]: Traceback (most recent call last):
Apr 09 10:48:37 celery-0.bi.privat watchmedo[6450]: File "/opt/galaxy/venv/lib/python3.11/site-packages/botocore/retryhandler.py", line 307, in _should_retry
Apr 09 10:48:37 celery-0.bi.privat watchmedo[6450]: return self._checker(
Apr 09 10:48:37 celery-0.bi.privat watchmedo[6450]: ^^^^^^^^^^^^^^
Apr 09 10:48:37 celery-0.bi.privat watchmedo[6450]: File "/opt/galaxy/venv/lib/python3.11/site-packages/botocore/retryhandler.py", line 363, in __call__
Apr 09 10:48:37 celery-0.bi.privat watchmedo[6450]: checker_response = checker(
Apr 09 10:48:37 celery-0.bi.privat watchmedo[6450]: ^^^^^^^^
Apr 09 10:48:37 celery-0.bi.privat watchmedo[6450]: File "/opt/galaxy/venv/lib/python3.11/site-packages/botocore/retryhandler.py", line 247, in __call__
Apr 09 10:48:37 celery-0.bi.privat watchmedo[6450]: return self._check_caught_exception(
Apr 09 10:48:37 celery-0.bi.privat watchmedo[6450]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Apr 09 10:48:37 celery-0.bi.privat watchmedo[6450]: File "/opt/galaxy/venv/lib/python3.11/site-packages/botocore/retryhandler.py", line 416, in _check_caught_exception
Apr 09 10:48:37 celery-0.bi.privat watchmedo[6450]: raise caught_exception
Apr 09 10:48:37 celery-0.bi.privat watchmedo[6450]: File "/opt/galaxy/venv/lib/python3.11/site-packages/botocore/endpoint.py", line 279, in _do_get_response
Apr 09 10:48:37 celery-0.bi.privat watchmedo[6450]: http_response = self._send(request)
Apr 09 10:48:37 celery-0.bi.privat watchmedo[6450]: ^^^^^^^^^^^^^^^^^^^
Apr 09 10:48:37 celery-0.bi.privat watchmedo[6450]: File "/opt/galaxy/venv/lib/python3.11/site-packages/botocore/endpoint.py", line 383, in _send
Apr 09 10:48:37 celery-0.bi.privat watchmedo[6450]: return self.http_session.send(request)
Apr 09 10:48:37 celery-0.bi.privat watchmedo[6450]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Apr 09 10:48:37 celery-0.bi.privat watchmedo[6450]: File "/opt/galaxy/venv/lib/python3.11/site-packages/botocore/httpsession.py", line 499, in send
Apr 09 10:48:37 celery-0.bi.privat watchmedo[6450]: raise ConnectTimeoutError(endpoint_url=request.url, error=e)
Apr 09 10:48:37 celery-0.bi.privat watchmedo[6450]: botocore.exceptions.ConnectTimeoutError: Connect timeout on endpoint URL: "https://usegalaxy-esg-api.test.fedcloud.eu:443/usegalaxy-esg-byos-minio-test"
Apr 09 10:48:37 celery-0.bi.privat watchmedo[6450]: [2025-04-09 10:48:37,550: DEBUG/main] Retry needed, action of: 5.873247782195808
```