MagicStack / MagicStack/asyncpg

Potential starvation with connection pool

未关闭
#1,247 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

还没有人认领这个 Issue。

主要语言
Python
星标
8.1k
派生
468
PR 合并指标
30 天内没有已合并 PR

描述

I have the following piece of code where I initialize 100 workers and a connection pool of size 10. Each worker loops and trys to acquire a connection and hold it for 200ms.

import asyncio

import asyncpg


async def worker(pool: asyncpg.Pool):
    while True:
        async with pool.acquire(timeout=10):
            # hold the connection for 200ms
            await asyncio.sleep(0.2)


async def main():
    async with asyncpg.pool.create_pool(max_size=10) as pool:
        async with asyncio.TaskGroup() as tg:
            for _ in range(100):
                tg.create_task(worker(pool))


asyncio.run(main())

When running on my laptop, this code reliably crashes with a TimeoutError after a few minutes, meaning that at least one worker failed to acquire a connection after 10 seconds. Given that there're 10 workers per connection and each worker holds the connection for 200ms, I would expect workers to be able to acquire a connection every 2 seconds.

This looks like starvation to me. Are there fairness guarantees regarding connection pools? Is there anything I could do other than just increasing the max pool size?

贡献指南

这个仓库没有索引到贡献指南

从这里开始

  1. 先读完整个 Issue,再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库,在一个分支上完成修改。
  4. 提交 Pull Request,并在描述里引用这个 Issue 编号。

调研方向

首先,使用 asyncio.TaskGroup、100 个 workers 和 asyncpg.pool.create_pool(max_size=10) 运行提供的 Python 复现程序。检查 pool.acquire(timeout=10) 在竞争下的行为,并确定观察到的 TimeoutError 是反映了饥饿,还是预期的调度;完成标准是记录 fairness guarantee,或识别出一个可复现的 pool 缺陷。

由索引模型根据 Issue 内容生成。

评估

技术栈
postgresql, python
领域
databases
Issue 类型
缺陷
难度
4/5
预计耗时
3-5 天
活跃度
停滞
描述清晰度
基本清楚
新手友好度
35/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。