lablup / lablup/backend.ai

Harden Standalone/Sentinel reconnection logic

Open
#10,753 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
670
Forks
183
Avg merge
17h 7m
Merged PRs (30d)
358

Description

## Objective

Strengthen the reconnection logic for both Standalone and Sentinel modes so that recovery succeeds even in abnormal termination scenarios such as socket deletion or container restarts.

## Details

- ValkeyStandaloneClient.need_reconnect() currently only checks if client is None — improve to verify actual connection state
- Accurately reflect GlideClient internal connection state
- In Sentinel mode, detect connection loss beyond just master address changes
- Ensure reconnection succeeds after socket file deletion under /tmp

## Reproduction Scenarios

- Stop and restart the Redis/Valkey container
- Delete the Valkey socket file under /tmp, then restart the container
- Scenario where only the operation client connection is lost

## Code References

- ValkeyStandaloneClient: src/ai/backend/common/clients/valkey_client/client.py (~line 133)
- ValkeyStandaloneClient.need_reconnect(): same file — currently returns client is None
- ValkeySentinelClient: same file (~line 226)
- ValkeySentinelClient.need_reconnect() and _get_master_address(): same file (~line 327, 348)
- MonitoringValkeyClient._reconnect(): same file (~line 551)

## Acceptance Criteria

- need_reconnect() accurately detects broken connections beyond client is None
- Reconnection succeeds after socket deletion + container restart
- Verified in both Standalone and Sentinel modes
- Tests covering each reproduction scenario are included

JIRA Issue: BA-5576

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.