lablup / lablup/backend.ai

Improve Valkey client reconnection stability and failure detection

Open
#10,751 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
670
Forks
183
Avg merge
17h 7m
Merged PRs (30d)
358

Description

## Background

The current MonitoringValkeyClient separates operation and monitor clients, but in certain failure scenarios the operation client becomes disconnected and does not recover. This epic aims to improve failure detection and automatic recovery across the Valkey client infrastructure.

## Current Architecture

MonitoringValkeyClient wraps two separate client instances for split responsibility:

- Operation client: Handles actual business operations with a user-specified request timeout. May perform long-running operations such as stream reads.
- Monitor client: A separate instance dedicated to health monitoring with a fixed 3-second timeout. Runs a background monitor task that pings every 10 seconds.

This separation ensures that long-running operations on the operation client do not block health monitoring. Both Standalone mode (ValkeyStandaloneClient via GlideClient) and Sentinel mode (ValkeySentinelClient with master discovery) are supported.

## Root Cause

- The operation client can become broken independently of the monitor client — it may time out or lose its connection while the monitor client continues to ping successfully.
- The current _check_connection() only pings the monitor client, so it has no way to detect operation client failures.
- ValkeyStandaloneClient.need_reconnect() only checks if client is None, which does not reflect actual connection state (e.g., after socket deletion).
- Since the monitor client stays healthy, _reconnect() (which tears down and reconnects both clients) is never triggered, leaving the operation client permanently broken.

## Affected Components

- Core client: MonitoringValkeyClient, ValkeyStandaloneClient, ValkeySentinelClient (src/ai/backend/common/clients/valkey_client/client.py)
- 14 domain-specific clients: valkey_stream, valkey_session, valkey_bgtask, valkey_stat, valkey_live, valkey_image, valkey_rate_limit, valkey_artifact, valkey_leader, valkey_container_log, valkey_schedule, valkey_artifact_registries, valkey_volume_stats, valkey_cache
- Manager dependency: src/ai/backend/manager/dependencies/infrastructure/redis.py (10 clients)
- Agent dependency: src/ai/backend/agent/dependencies/infrastructure/redis.py (4 clients)
- Health checker: src/ai/backend/common/health_checker/checkers/valkey.py

## Sub-issue Execution Order

1. BA-5578: Apply async-with pattern with retry-based disconnection — establishes the foundation for usage-time failure tracking
1. BA-5576: Harden Standalone/Sentinel reconnection logic — ensures reconnection actually works once failures are detected
1. BA-5577: Handle operation client recovery when monitor client is healthy — addresses the core scenario and exposes status via internal health API

## Scope

- All sub-issues must be verified in both Standalone and Sentinel modes
- Each sub-issue includes relevant test coverage

## Future Considerations

- Apply the same reconnection patterns to the WebUI connection pool

JIRA Issue: BA-5574

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.