MagicStack / MagicStack/asyncpg
ConnectionError from _cancel() during CancelledError not caught, crashes callers
还没有人认领这个 Issue。
- 主要语言
- Python
- 星标
- 8.1k
- 派生
- 468
- PR 合并指标
- 30 天内没有已合并 PR
描述
## Environment
- **asyncpg version**: 0.31.0 (also reproduced on 0.30.x)
- **PostgreSQL version**: 16
- **Python version**: 3.11.14
- **Platform**: Linux (Kubernetes)
- **pgbouncer**: No
- **SQLAlchemy**: 2.0.23
## Summary
When an asyncpg operation is cancelled via `asyncio.CancelledError` while mid-query, the cancellation mechanism in `connect_utils._cancel` can raise a **built-in `ConnectionError`** that escapes to the caller. This is problematic because:
1. Callers (e.g. SQLAlchemy) expect asyncpg-specific exception types and don't handle built-in `ConnectionError`
2. The cancel operation is inherently best-effort — if the cancel connection fails, the error should be suppressed or wrapped, not propagated
This is related to #1211 but occurs on **non-`direct_tls`** connections via the cancel request code path.
## Reproduction flow
1. An asyncpg connection is executing a query (e.g. inside SQLAlchemy's `session.execute()`)
2. The asyncio task is cancelled (`task.cancel()`)
3. `CancelledError` propagates into `protocol.query()` / `bind_execute`
4. asyncpg's cancellation handler tries to send a PostgreSQL cancel request by opening a **new** SSL connection via `connect_utils._cancel` → `_create_ssl_connection`
5. The new connection fails (server already closed the original, or network issue)
6. `TLSUpgradeProto.connection_lost()` raises built-in `ConnectionError('unexpected connection_lost() call')`
7. This escapes through `connect_utils._cancel` (which has **no error handling** around `_create_ssl_connection`)
8. Caller receives `ConnectionError` instead of `CancelledError`
## Traceback
```
asyncio.exceptions.CancelledError (original exception)
During handling of the above exception, another exception occurred:
File "asyncpg/transaction.py", line 206, in __rollback
await self._connection.execute(query)
File "asyncpg/connection.py", line 350, in execute
result = await self._protocol.query(query, timeout)
File "asyncpg/connection.py", line 1584, in _cancel
await connect_utils._cancel(
File "asyncpg/connect_utils.py", line 1040, in _cancel
tr, pr = await _create_ssl_connection(
File "asyncpg/connect_utils.py", line 752, in _create_ssl_connection
do_ssl_upgrade = await pr.on_data
^^^^^^^^^^^^^^^^
ConnectionError: unexpected connection_lost() call
```
## Root cause
Two issues in `connect_utils.py`:
### 1. `_cancel()` has no error handling around `_create_ssl_connection`
```python
async def _cancel(*, loop, addr, params, backend_pid, backend_secret):
...
if params.ssl and params.sslmode != SSLMode.allow:
tr, pr = await _create_ssl_connection(...) # ← no try/except!
...
```
The cancel request is best-effort (we're telling PostgreSQL to cancel a query on a connection that may already be dead). If opening the cancel connection fails, the error should be suppressed or wrapped in `asyncpg.InterfaceError`, not propagated as a raw `ConnectionError`.
### 2. `TLSUpgradeProto.connection_lost()` raises built-in `ConnectionError`
```python
def connection_lost(self, exc):
if not self.on_data.done():
if exc is None:
exc = ConnectionError('unexpected connection_lost() call')
self.on_data.set_exception(exc)
```
This raises a **built-in Python `ConnectionError`**, not an asyncpg exception type. Callers like SQLAlchemy check for `asyncpg.InterfaceError` or `asyncpg.PostgresError` to detect disconnects. A built-in `ConnectionError` bypasses all those checks, which means:
- SQLAlchemy's `is_disconnect()` doesn't recognize it
- SQLAlchemy's pool pre-ping handler (`_do_ping_w_event`) only catches `self.loaded_dbapi.Error`, so `ConnectionError` escapes
- The pool's retry logic (which would create a fresh connection) never triggers
## Suggested fix
**Option A** (minimal): Catch `OSError` (parent of `ConnectionError`) in `connect_utils._cancel()` and suppress it — cancel is best-effort:
```python
async def _cancel(*, loop, addr, params, backend_pid, backend_secret):
...
try:
if params.ssl and params.sslmode != SSLMode.allow:
tr, pr = await _create_ssl_connection(...)
...
except OSError:
# Cancel is best-effort. If we can't reach the server, the
# connection is dead anyway.
return
```
**Option B** (comprehensive): Also change `TLSUpgradeProto.connection_lost()` to raise `asyncpg.InterfaceError` instead of built-in `ConnectionError`, so callers can handle it consistently:
```python
def connection_lost(self, exc):
if not self.on_data.done():
if exc is None:
exc = InterfaceError('unexpected connection_lost() call')
self.on_data.set_exception(exc)
```
## Impact
This causes process crashes in production services. When a task is cancelled during a DB query, the `ConnectionError` escapes all exception handlers (which expect either `CancelledError` or asyncpg-specific exceptions) and terminates the process.
This is 100% correlated with `CancelledError` in our logs — every `ConnectionError: unexpected connection_lost()` we've seen is triggered by task cancellation.
## Additional context
We use Google CloudSQL with SSL connections. The PostgreSQL server is accessed over SSL (non-`direct_tls`), which means the cancel code path goes through `_create_ssl_connection` to establish a new SSL connection for sending the cancel request.
贡献指南
这个仓库没有索引到贡献指南
从这里开始
- 先读完整个 Issue,再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库,在一个分支上完成修改。
- 提交 Pull Request,并在描述里引用这个 Issue 编号。
调研方向
从 connect_utils.py 中的 _cancel 和 _create_ssl_connection 开始,然后检查 TLSUpgradeProto.connection_lost(),以了解取消失败是如何转换的。为 SSL 取消连接失败添加一个回归测试,防止原始的 ConnectionError 传递到调用方,然后运行相关的取消和连接测试。
由索引模型根据 Issue 内容生成。
评估
- 技术栈
- postgresql, python
- 领域
- databases
- Issue 类型
- 缺陷
- 难度
- 4/5
- 预计耗时
- 3-5 天
- 活跃度
- 冷清
- 描述清晰度
- 基本清楚
- 新手友好度
- 55/100