MagicStack / MagicStack/asyncpg
Potential starvation with connection pool
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 8.1k
- Forks
- 468
- PR merge metrics
- No merged PRs in 30d
Description
I have the following piece of code where I initialize 100 workers and a connection pool of size 10. Each worker loops and trys to acquire a connection and hold it for 200ms.
```python
import asyncio
import asyncpg
async def worker(pool: asyncpg.Pool):
while True:
async with pool.acquire(timeout=10):
# hold the connection for 200ms
await asyncio.sleep(0.2)
async def main():
async with asyncpg.pool.create_pool(max_size=10) as pool:
async with asyncio.TaskGroup() as tg:
for _ in range(100):
tg.create_task(worker(pool))
asyncio.run(main())
```
When running on my laptop, this code reliably crashes with a `TimeoutError` after a few minutes, meaning that at least one worker failed to acquire a connection after 10 seconds. Given that there're 10 workers per connection and each worker holds the connection for 200ms, I would expect workers to be able to acquire a connection every 2 seconds.
This looks like starvation to me. Are there fairness guarantees regarding connection pools? Is there anything I could do other than just increasing the max pool size?
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by running the provided Python reproduction with asyncio.TaskGroup, 100 workers, and asyncpg.pool.create_pool(max_size=10). Inspect the behavior of pool.acquire(timeout=10) under contention and determine whether the observed TimeoutError reflects starvation or expected scheduling; done means documenting the fairness guarantee or identifying a reproducible pool defect.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- postgresql, python
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100