Improve Valkey client reconnection stability and failure detection
- Dominant language
- Python
- Stars
- 670
- Forks
- 183
- Avg merge
- 17h 7m
- Merged PRs (30d)
- 358
Description
## Background
The current MonitoringValkeyClient separates operation and monitor clients, but in certain failure scenarios the operation client becomes disconnected and does not recover. This epic aims to improve failure detection and automatic recovery across the Valkey client infrastructure.
## Current Architecture
MonitoringValkeyClient wraps two separate client instances for split responsibility:
- Operation client: Handles actual business operations with a user-specified request timeout. May perform long-running operations such as stream reads.
- Monitor client: A separate instance dedicated to health monitoring with a fixed 3-second timeout. Runs a background monitor task that pings every 10 seconds.
This separation ensures that long-running operations on the operation client do not block health monitoring. Both Standalone mode (ValkeyStandaloneClient via GlideClient) and Sentinel mode (ValkeySentinelClient with master discovery) are supported.
## Root Cause
- The operation client can become broken independently of the monitor client — it may time out or lose its connection while the monitor client continues to ping successfully.
- The current _check_connection() only pings the monitor client, so it has no way to detect operation client failures.
- ValkeyStandaloneClient.need_reconnect() only checks if client is None, which does not reflect actual connection state (e.g., after socket deletion).
- Since the monitor client stays healthy, _reconnect() (which tears down and reconnects both clients) is never triggered, leaving the operation client permanently broken.
## Affected Components
- Core client: MonitoringValkeyClient, ValkeyStandaloneClient, ValkeySentinelClient (src/ai/backend/common/clients/valkey_client/client.py)
- 14 domain-specific clients: valkey_stream, valkey_session, valkey_bgtask, valkey_stat, valkey_live, valkey_image, valkey_rate_limit, valkey_artifact, valkey_leader, valkey_container_log, valkey_schedule, valkey_artifact_registries, valkey_volume_stats, valkey_cache
- Manager dependency: src/ai/backend/manager/dependencies/infrastructure/redis.py (10 clients)
- Agent dependency: src/ai/backend/agent/dependencies/infrastructure/redis.py (4 clients)
- Health checker: src/ai/backend/common/health_checker/checkers/valkey.py
## Sub-issue Execution Order
1. BA-5578: Apply async-with pattern with retry-based disconnection — establishes the foundation for usage-time failure tracking
1. BA-5576: Harden Standalone/Sentinel reconnection logic — ensures reconnection actually works once failures are detected
1. BA-5577: Handle operation client recovery when monitor client is healthy — addresses the core scenario and exposes status via internal health API
## Scope
- All sub-issues must be verified in both Standalone and Sentinel modes
- Each sub-issue includes relevant test coverage
## Future Considerations
- Apply the same reconnection patterns to the WebUI connection pool
JIRA Issue: BA-5574
Contributor guide
Assessment
This issue has not been assessed yet.