Fix spurious error logs from Valkey connection monitor during graceful shutdown
- Dominant language
- Python
- Stars
- 670
- Forks
- 183
- Avg merge
- 21h 49m
- Merged PRs (30d)
- 404
Description
## Problem Description
When shutting down the manager server, spurious ERROR level logs are emitted from the Valkey connection monitor task:
```
ERROR ai.backend.common.clients.valkey_client.client Error in Valkey connection monitor:
Traceback (most recent call last):
File ".../client.py", line 536, in _monitor_connection
await asyncio.sleep(_DEFAULT_MONITOR_INTERVAL)
asyncio.exceptions.CancelledError
```
This is followed by timeout warnings from aiotools:
```
WARNING aiotools.server Timeout during waiting for child processes; killing all
WARNING aiotools.fork Force-killed hanging child: 88863
```
## Root Cause
In `MonitoringValkeyClient._monitor_connection()`, the exception handler catches `BaseException` which includes `asyncio.CancelledError`. During graceful shutdown:
1. aiotools server framework propagates cancellation to all tasks
1. The `_monitor_connection` task receives `CancelledError` before `disconnect()` is called
1. At this point, `self._closed` is still `False`
1. The error is logged even though cancellation is a normal shutdown scenario
## Solution
- Handle `asyncio.CancelledError` explicitly and re-raise without logging
- Change `except BaseException` to `except Exception` to exclude `CancelledError`
- Fix a bug in the finally block log message format
## Files Changed
- `src/ai/backend/common/clients/valkey_client/client.py`
JIRA Issue: BA-3593
Contributor guide
Research direction
Start in src/ai/backend/common/clients/valkey_client/client.py at MonitoringValkeyClient._monitor_connection(). Review the cancellation path and the finally-block log formatting, then exercise graceful manager-server shutdown. Done means normal task cancellation no longer emits the spurious ERROR log or causes the reported shutdown warnings.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 58/100