MagicStack / MagicStack/asyncpg
Potential starvation with connection pool
还没有人认领这个 Issue。
- 主要语言
- Python
- 星标
- 8.1k
- 派生
- 468
- PR 合并指标
- 30 天内没有已合并 PR
描述
I have the following piece of code where I initialize 100 workers and a connection pool of size 10. Each worker loops and trys to acquire a connection and hold it for 200ms.
import asyncio
import asyncpg
async def worker(pool: asyncpg.Pool):
while True:
async with pool.acquire(timeout=10):
# hold the connection for 200ms
await asyncio.sleep(0.2)
async def main():
async with asyncpg.pool.create_pool(max_size=10) as pool:
async with asyncio.TaskGroup() as tg:
for _ in range(100):
tg.create_task(worker(pool))
asyncio.run(main())
When running on my laptop, this code reliably crashes with a TimeoutError after a few minutes, meaning that at least one worker failed to acquire a connection after 10 seconds. Given that there're 10 workers per connection and each worker holds the connection for 200ms, I would expect workers to be able to acquire a connection every 2 seconds.
This looks like starvation to me. Are there fairness guarantees regarding connection pools? Is there anything I could do other than just increasing the max pool size?
贡献指南
这个仓库没有索引到贡献指南
从这里开始
- 先读完整个 Issue,再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库,在一个分支上完成修改。
- 提交 Pull Request,并在描述里引用这个 Issue 编号。
调研方向
首先,使用 asyncio.TaskGroup、100 个 workers 和 asyncpg.pool.create_pool(max_size=10) 运行提供的 Python 复现程序。检查 pool.acquire(timeout=10) 在竞争下的行为,并确定观察到的 TimeoutError 是反映了饥饿,还是预期的调度;完成标准是记录 fairness guarantee,或识别出一个可复现的 pool 缺陷。
由索引模型根据 Issue 内容生成。
评估
- 技术栈
- postgresql, python
- 领域
- databases
- Issue 类型
- 缺陷
- 难度
- 4/5
- 预计耗时
- 3-5 天
- 活跃度
- 停滞
- 描述清晰度
- 基本清楚
- 新手友好度
- 35/100