lablup / lablup/backend.ai

Fix spurious error logs from Valkey connection monitor during graceful shutdown

Open
#7,629 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
670
Forks
183
Avg merge
21h 49m
Merged PRs (30d)
404

Description

## Problem Description

When shutting down the manager server, spurious ERROR level logs are emitted from the Valkey connection monitor task:

```
ERROR ai.backend.common.clients.valkey_client.client Error in Valkey connection monitor:
Traceback (most recent call last):
File ".../client.py", line 536, in _monitor_connection
await asyncio.sleep(_DEFAULT_MONITOR_INTERVAL)
asyncio.exceptions.CancelledError
```

This is followed by timeout warnings from aiotools:

```
WARNING aiotools.server Timeout during waiting for child processes; killing all
WARNING aiotools.fork Force-killed hanging child: 88863
```

## Root Cause

In `MonitoringValkeyClient._monitor_connection()`, the exception handler catches `BaseException` which includes `asyncio.CancelledError`. During graceful shutdown:

1. aiotools server framework propagates cancellation to all tasks
1. The `_monitor_connection` task receives `CancelledError` before `disconnect()` is called
1. At this point, `self._closed` is still `False`
1. The error is logged even though cancellation is a normal shutdown scenario

## Solution

- Handle `asyncio.CancelledError` explicitly and re-raise without logging
- Change `except BaseException` to `except Exception` to exclude `CancelledError`
- Fix a bug in the finally block log message format

## Files Changed

- `src/ai/backend/common/clients/valkey_client/client.py`

JIRA Issue: BA-3593

Contributor guide

Open the contributing guide

Research direction

Start in src/ai/backend/common/clients/valkey_client/client.py at MonitoringValkeyClient._monitor_connection(). Review the cancellation path and the finally-block log formatting, then exercise graceful manager-server shutdown. Done means normal task cancellation no longer emits the spurious ERROR log or causes the reported shutdown warnings.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.