MagicStack / MagicStack/asyncpg

Potential starvation with connection pool

Open
#1,247 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
8.1k
Forks
468
PR merge metrics
No merged PRs in 30d

Description

I have the following piece of code where I initialize 100 workers and a connection pool of size 10. Each worker loops and trys to acquire a connection and hold it for 200ms.

```python
import asyncio

import asyncpg

async def worker(pool: asyncpg.Pool):
while True:
async with pool.acquire(timeout=10):
# hold the connection for 200ms
await asyncio.sleep(0.2)

async def main():
async with asyncpg.pool.create_pool(max_size=10) as pool:
async with asyncio.TaskGroup() as tg:
for _ in range(100):
tg.create_task(worker(pool))

asyncio.run(main())
```

When running on my laptop, this code reliably crashes with a `TimeoutError` after a few minutes, meaning that at least one worker failed to acquire a connection after 10 seconds. Given that there're 10 workers per connection and each worker holds the connection for 200ms, I would expect workers to be able to acquire a connection every 2 seconds.

This looks like starvation to me. Are there fairness guarantees regarding connection pools? Is there anything I could do other than just increasing the max pool size?

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by running the provided Python reproduction with asyncio.TaskGroup, 100 workers, and asyncpg.pool.create_pool(max_size=10). Inspect the behavior of pool.acquire(timeout=10) under contention and determine whether the observed TimeoutError reflects starvation or expected scheduling; done means documenting the fairness guarantee or identifying a reproducible pool defect.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgresql, python
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.