pool: not closing connections
- Linguagem predominante
- Python
- Estrelas
- 1.4k
- Forks
- 170
- Métricas de merge de PRs
- Nenhum PR com merge em 30d
Descrição
Creating a connection pool with `minsize=1, maxsize=10`, all opened connections remain open after they are used and returned.
If we have no DB activity, then we should maintain only `minsize` open connections, not more.
I've created the following script that reproduces the issue, what this does is:
- have N tasks (per cli argument)
- sequentially opens a lot of connections/cursors on each task
- constantly counts the `pool.size`
- prints a summary at the end of the tasks
```python
import asyncio
import os
import aiopg
import sys
default_dsn = 'dbname=postgres user=postgres password=postgres host=localhost'
dsn = os.getenv('DSN', default_dsn)
pool = None
creating_pool = False
tasks = {}
@asyncio.coroutine
def get_pool():
global pool, creating_pool
if creating_pool and pool is None:
yield from asyncio.sleep(0.1)
return (yield from get_pool())
if pool is None:
creating_pool = True
print('create a pool')
pool = yield from aiopg.create_pool(
dsn,
maxsize=10,
minsize=1,
application_name='testpool'
)
creating_pool = False
return pool
min_size = None
max_size = None
current_size = None
@asyncio.coroutine
def check_size():
global current_size, max_size, min_size, pool
i = 0
while True:
yield from asyncio.sleep(0.01)
i += 1
if pool is None:
continue
size = pool.size
if current_size is None or size != current_size:
current_size = size
if max_size is None or size > max_size:
max_size = size
if min_size is None or size < min_size:
min_size = size
if len(tasks) == 0:
print(i, 'count iterations')
# wait 5 seconds, check size every second
#
for x in range(1, 5):
print(' actual size: ', pool.size)
yield from asyncio.sleep(1)
print('SUMMARY')
print(' min size seen:', min_size)
print(' max size seen:', max_size)
print(' final size:', pool.size)
break
@asyncio.coroutine
def do_select(total, key=0):
tasks[key] = total
pool = yield from get_pool()
for i in range(total):
with (yield from pool) as conn:
cur = yield from conn.cursor()
yield from cur.execute("SELECT 1")
ret = yield from cur.fetchone()
assert ret == (1,)
tasks.pop(key)
def main():
loop = asyncio.get_event_loop()
tasks = [check_size()] + [
# execute do_select() n times
do_select(total=100, key=x) for x in range(int(sys.argv[1]))
]
print(len(tasks), 'tasks')
loop.run_until_complete(asyncio.wait(tasks))
if __name__ == '__main__':
main()
```
execute like the following:
```
python testpool.py 1
```
And will produce something like this:
```
2 tasks
create a pool
6 count iterations
actual size: 1
actual size: 1
actual size: 1
actual size: 1
SUMMARY
min size seen: 1
max size seen: 1
final size: 1
```
That's fine, a single db task requests cursors sequentially, so a single db connection is used. min, max and final size are all 1.
The problem is when we have multiple concurrent tasks:
```
python testpool.py 1000
1001 tasks
create a pool
3343 count iterations
actual size: 10
actual size: 10
actual size: 10
actual size: 10
SUMMARY
min size seen: 1
max size seen: 10
final size: 10
```
In this case, 1000 concurrent tasks are sharing the connection pool, they use up to `maxsize` connections and thats ok, but the problem is that at the end of the script it is still maintaining 10 open connections and we should have just 1.
Guia de contribuição
Avaliação
Esta issue ainda não foi avaliada.