MagicStack / MagicStack/asyncpg
Asyncpg does not seem to clear memory allocated to Pool object after db.disconnect() (possible memory leak)
Dieses Issue hat noch niemand übernommen.
- Vorherrschende Sprache
- Python
- Sterne
- 8.1k
- Forks
- 468
- PR-Merge-Kennzahlen
- Keine gemergten PRs in 30 T.
Beschreibung
* **asyncpg version**: 0.25.0 (tried with 0.23.0 and 0.24.0)
* **PostgreSQL version**: 11.16
* **Do you use a PostgreSQL SaaS? If so, which? Can you reproduce
the issue with a local PostgreSQL install?**: No, using the postgres:11.16 docker image directrly
* **Python version**: 3.10.5 (+ latest 3.9 and 3.8)
* **Platform**: Ubuntu 64 and docker
* **Do you use pgbouncer?**: No
* **Did you install asyncpg with pip?**: Yes
* **If you built asyncpg locally, which version of Cython did you use?**: N/A
* **Can the issue be reproduced under both asyncio and
[uvloop](https://github.com/magicstack/uvloop)?**: I only tried asyncio
We observed PODs running out of memory.
They are reading from a queue constantly and calling this function
```python
async def process_request(message, db : ):
await db.connect()
# do_something(message, db)
await db.disconnect()
```
I am confident this is related to asyncpg as here are some of the objects that are created in memory with every iteration of connect/disconnect.
```
asyncpg.pgproto.pgproto.ReadBuffer | 1000 | 132.81 KB
collections.OrderedDict | 1000 | 125.00 KB
asyncpg.pool.PoolConnectionHolder | 1000 | 117.19 KB
asyncio.events.TimerHandle | 1001 | 109.48 KB
```
I have tried:
- waiting for the TimerHandle to expire (60s)
- calling engine.dispose() manually
- calling the garbage collector manually
Sometime in the past we encountered a problem that made 10000s of connections in a few seconds (crashing the DB) that was fixed by abandoning this connect/disconnect pattern, so maybe these two are related.
The db is initialised as such
```python
databases.Database(db_uri)
```
This problem does not exists in the sqlite driver.
Beitragsleitfaden
Für dieses Repository ist kein Beitragsleitfaden indexiert
Erste Schritte
- Lies das ganze Issue und danach den Beitragsleitfaden des Projekts.
- Schreib ins Issue, dass du es übernimmst — das erspart doppelte Arbeit.
- Forke das Repository und arbeite in einem Branch.
- Öffne einen Pull Request, der die Issue-Nummer nennt.
Rechercherichtung
Start by reproducing the repeated db.connect() and db.disconnect() pattern described with asyncpg 0.25.0, PostgreSQL 11.16, Python 3.10.5, and the postgres:11.16 Docker image. Inspect whether asyncpg.pgproto.pgproto.ReadBuffer, OrderedDict, PoolConnectionHolder, and asyncio.events.TimerHandle objects remain after disconnect and garbage collection. Compare the behavior with the sqlite driver and determine whether the retained allocations represent a leak.
Vom Indexierungsmodell aus dem Issue-Text verfasst.
Bewertung
- Tech-Stack
- postgresql, python
- Bereich
- backend, databases
- Issue-Typ
- Bug
- Schwierigkeit
- 4/5
- Geschätzter Aufwand
- 3-5 Tage
- Aktivitätsstatus
- Veraltet
- Klarheit
- Muss geklärt werden
- Anfängerfreundlichkeit
- 25/100