Redis Events Plugin - can become permanently disconnected from its redis cluster

We use the Redis Events plugin with an acapy mediator hosted on OpenShift.

Observations:

• When the mediator connects to the redis cluster (6 nodes) the connection is brokered through the associated service in front of the redis cluster, however once the connection is established the connection appears to be a direct connection with the nodes on the cluster. i.e. The connected IPs are within the subnet of the redis nodes, rather than the subnet of the service. This sets up the condition for the remaining observations.
• When the nodes within the redis cluster's StatefulSet are recreated they are not guaranteed to come back up with the same IP address.
• When the IP(s) through which the agent is connected to the redis cluster are no longer associated with a redis node the plugin starts throwing errors - see the example below.
• Note - I had to trigger the statefulset nodes to be recreated twice before I observed any errors.
• The mediator, at least in my testing, appears to continue to function during this time.
• The connection errors will continue until the mediator agent is restarted, at which time a new connection is established with the cluster by the plugin.

Example connection error:

2025-07-24 12:28:04,962 redis_events.v1_0.redis_queue.events ERROR Failed to process and send webhook, Redis Cluster cannot be connected. Please provide at least one reachable node: Error 113 connecting to 10.97.23.173:6379. 113.
Traceback (most recent call last):
  File "/home/aries/.local/lib/python3.9/site-packages/redis/asyncio/connection.py", line 577, in connect
    await self.retry.call_with_retry(
  File "/home/aries/.local/lib/python3.9/site-packages/redis/asyncio/retry.py", line 59, in call_with_retry
    return await do()
  File "/home/aries/.local/lib/python3.9/site-packages/redis/asyncio/connection.py", line 922, in _connect
    reader, writer = await asyncio.open_connection(
  File "/usr/local/lib/python3.9/asyncio/streams.py", line 52, in open_connection
    transport, _ = await loop.create_connection(
  File "/usr/local/lib/python3.9/asyncio/base_events.py", line 1065, in create_connection
    raise exceptions[0]
  File "/usr/local/lib/python3.9/asyncio/base_events.py", line 1050, in create_connection
    sock = await self._connect_sock(
  File "/usr/local/lib/python3.9/asyncio/base_events.py", line 961, in _connect_sock
    await self.sock_connect(sock, address)
  File "/usr/local/lib/python3.9/asyncio/selector_events.py", line 500, in sock_connect
    return await fut
  File "/usr/local/lib/python3.9/asyncio/futures.py", line 284, in __await__
    yield self  # This tells Task to wait for completion.
  File "/usr/local/lib/python3.9/asyncio/tasks.py", line 328, in __wakeup
    future.result()
  File "/usr/local/lib/python3.9/asyncio/futures.py", line 201, in result
    raise self._exception
  File "/usr/local/lib/python3.9/asyncio/selector_events.py", line 535, in _sock_connect_cb
    raise OSError(err, f'Connect call failed {address}')
OSError: [Errno 113] Connect call failed ('10.97.23.173', 6379)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/aries/.local/lib/python3.9/site-packages/redis/asyncio/cluster.py", line 1197, in initialize
    if not (await startup_node.execute_command("INFO")).get(
  File "/home/aries/.local/lib/python3.9/site-packages/redis/asyncio/cluster.py", line 1016, in execute_command
    await connection.send_packed_command(connection.pack_command(*args), False)
  File "/home/aries/.local/lib/python3.9/site-packages/redis/asyncio/connection.py", line 713, in send_packed_command
    await self.connect()
  File "/home/aries/.local/lib/python3.9/site-packages/redis/asyncio/connection.py", line 585, in connect
    raise ConnectionError(self._error_message(e))
redis.exceptions.ConnectionError: Error 113 connecting to 10.97.23.173:6379. 113.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/aries/.local/lib/python3.9/site-packages/redis_events/v1_0/redis_queue/events/__init__.py", line 126, in handle_event
    await redis.rpush(
  File "/home/aries/.local/lib/python3.9/site-packages/redis/asyncio/cluster.py", line 676, in execute_command
    await self.initialize()
  File "/home/aries/.local/lib/python3.9/site-packages/redis/asyncio/cluster.py", line 383, in initialize
    await self.nodes_manager.initialize()
  File "/home/aries/.local/lib/python3.9/site-packages/redis/asyncio/cluster.py", line 1291, in initialize
    raise RedisClusterException(
redis.exceptions.RedisClusterException: Redis Cluster cannot be connected. Please provide at least one reachable node: Error 113 connecting to 10.97.23.173:6379. 113.

Start at redis_events.v1_0.redis_queue.events, especially handle_event, and inspect how RedisCluster initialization behaves after the cluster advertises unreachable node IPs. Reproduce the failure by recreating the Redis StatefulSet nodes and observe whether the plugin recovers without restarting the mediator. Done means webhook processing resumes after node replacement, with regression coverage for the recovery path.