MagicStack / MagicStack/asyncpg
Asyncpg does not seem to clear memory allocated to Pool object after db.disconnect() (possible memory leak)
Nadie ha tomado este issue todavía.
- Lenguaje dominante
- Python
- Estrellas
- 8.1k
- Forks
- 468
- Métricas de merge de PR
- Sin PR fusionados en 30 d
Descripción
* **asyncpg version**: 0.25.0 (tried with 0.23.0 and 0.24.0)
* **PostgreSQL version**: 11.16
* **Do you use a PostgreSQL SaaS? If so, which? Can you reproduce
the issue with a local PostgreSQL install?**: No, using the postgres:11.16 docker image directrly
* **Python version**: 3.10.5 (+ latest 3.9 and 3.8)
* **Platform**: Ubuntu 64 and docker
* **Do you use pgbouncer?**: No
* **Did you install asyncpg with pip?**: Yes
* **If you built asyncpg locally, which version of Cython did you use?**: N/A
* **Can the issue be reproduced under both asyncio and
[uvloop](https://github.com/magicstack/uvloop)?**: I only tried asyncio
We observed PODs running out of memory.
They are reading from a queue constantly and calling this function
```python
async def process_request(message, db : ):
await db.connect()
# do_something(message, db)
await db.disconnect()
```
I am confident this is related to asyncpg as here are some of the objects that are created in memory with every iteration of connect/disconnect.
```
asyncpg.pgproto.pgproto.ReadBuffer | 1000 | 132.81 KB
collections.OrderedDict | 1000 | 125.00 KB
asyncpg.pool.PoolConnectionHolder | 1000 | 117.19 KB
asyncio.events.TimerHandle | 1001 | 109.48 KB
```
I have tried:
- waiting for the TimerHandle to expire (60s)
- calling engine.dispose() manually
- calling the garbage collector manually
Sometime in the past we encountered a problem that made 10000s of connections in a few seconds (crashing the DB) that was fixed by abandoning this connect/disconnect pattern, so maybe these two are related.
The db is initialised as such
```python
databases.Database(db_uri)
```
This problem does not exists in the sqlite driver.
Guía de contribución
No hay ninguna guía de contribución indexada para este repositorio
Primeros pasos
- Lee el issue completo y luego la guía de contribución del proyecto.
- Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
- Haz un fork del repositorio y trabaja en una rama.
- Abre un pull request que haga referencia al número del issue.
Línea de trabajo
Start by reproducing the repeated db.connect() and db.disconnect() pattern described with asyncpg 0.25.0, PostgreSQL 11.16, Python 3.10.5, and the postgres:11.16 Docker image. Inspect whether asyncpg.pgproto.pgproto.ReadBuffer, OrderedDict, PoolConnectionHolder, and asyncio.events.TimerHandle objects remain after disconnect and garbage collection. Compare the behavior with the sqlite driver and determine whether the retained allocations represent a leak.
Escrito por el modelo de indexación a partir del texto del issue.
Evaluación
- Stack tecnológico
- postgresql, python
- Área
- backend, databases
- Tipo de issue
- Error
- Dificultad
- 4/5
- Tiempo estimado
- 3-5 días
- Estado de actividad
- Estancado
- Claridad
- Necesita aclaración
- Aptitud para principiantes
- 25/100