MagicStack / MagicStack/asyncpg

Handle additional InterfaceError types when inside a transaction

未关闭
#956 0 条评论 1 个 reaction 已指派 0 人 在 GitHub 查看

还没有人认领这个 Issue。

主要语言
Python
星标
8.1k
派生
468
PR 合并指标
30 天内没有已合并 PR

描述

  • asyncpg version: 0.26.0
  • PostgreSQL version: PostgreSQL 14.5 (Debian 14.5-1.pgdg110+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 10.2.1-6) 10.2.1 20210110, 64-bit - Also fails on older versions such as PG 13
  • Do you use a PostgreSQL SaaS? If so, which? Can you reproduce
    the issue with a local PostgreSQL install?
    : Google CloudSQL, but I'm reproducing this locally in a test.
  • Python version: 3.10
  • Platform: MacOS
  • Do you use pgbouncer?: No
  • Did you install asyncpg with pip?: yes
  • If you built asyncpg locally, which version of Cython did you use?: n/a
  • Can the issue be reproduced under both asyncio and
    uvloop?
    : yes

When an InterfaceError is raised during a transaction, asyncpg does not explicitly handle all variants of this exception, and raises it.

https://github.com/MagicStack/asyncpg/blob/5f908e679a6264c5fcf8a92895a2f34a9387e4da/asyncpg/transaction.py#L65-L79

For example, it does not handle the type asyncpg.exceptions.ConnectionDoesNotExistError - and as such, this leads to leftover connections while trying to close the pool. Simple reproduction that forces the error by setting idle_in_transaction_session_timeout to be very small - basically telling the server to close a connection before it can complete a query:

pool = await asyncpg.create_pool(
    dsn=dsn,
    min_size=1,
    max_size=4,
    server_settings={
        "idle_in_transaction_session_timeout": "1",
    },
)


async def query_gen():
    async with pool.acquire(timeout=5) as con:
        async with con.transaction(readonly=True):
            await con.fetch(
                "SELECT * FROM my_db.my_schema.my_table;", timeout=15
            )


tasks = []
for i in range(40):
    tasks.append(query_gen())
results = await asyncio.gather(*tasks, return_exceptions=True)

print(results, flush=True)

await pool.close()

The results List is all exceptions:

[InterfaceError('cannot call Transaction.__aexit__(): the underlying connection is closed'), InterfaceError('cannot call Transaction.__aexit__(): the underlying connection is closed'), InterfaceError('cannot call Transaction.__aexit__(): the underlying connection is closed'), InterfaceError('cannot call Transaction.__aexit__(): the underlying connection is closed'), ...]

The exception type is asyncpg.exceptions.ConnectionDoesNotExistError. And the pool.close() will print a warning:

asyncpg.pool:Pool.close() is taking over 60 seconds to complete. Check if you have any unreleased connections left. Use asyncio.wait_for() to set a timeout for Pool.close().

When this happens, I do believe the connection pool should set the _in_use attribute for the connection holder to False and / or delete the pool connection holder.

This does not happen when not using the async with con.transaction() context manager.

Note: Under normal conditions / default settings, I see this rarely, however, I do see the pool.close() warning on occasion.

贡献指南

这个仓库没有索引到贡献指南

从这里开始

  1. 先读完整个 Issue,再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库,在一个分支上完成修改。
  4. 提交 Pull Request,并在描述里引用这个 Issue 编号。

调研方向

从 asyncpg/transaction.py 中链接的第 65-79 行开始,然后运行提供的 asyncio 复现,并将 idle_in_transaction_session_timeout 设置为 1。跟踪事务期间的 ConnectionDoesNotExistError 如何影响池,并验证连接已释放,且 pool.close() 完成时不会出现未释放连接警告。

由索引模型根据 Issue 内容生成。

评估

技术栈
postgresql, python
领域
database
Issue 类型
缺陷
难度
3/5
预计耗时
1-2 天
活跃度
停滞
描述清晰度
基本清楚
新手友好度
38/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。