MagicStack / MagicStack/asyncpg

Asyncpg does not seem to clear memory allocated to Pool object after db.disconnect() (possible memory leak)

Offen
#929 1 Kommentar 5 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen

Dieses Issue hat noch niemand übernommen.

Vorherrschende Sprache
Python
Sterne
8.1k
Forks
468
PR-Merge-Kennzahlen
Keine gemergten PRs in 30 T.

Beschreibung

* **asyncpg version**: 0.25.0 (tried with 0.23.0 and 0.24.0)
* **PostgreSQL version**: 11.16
* **Do you use a PostgreSQL SaaS? If so, which? Can you reproduce
the issue with a local PostgreSQL install?**: No, using the postgres:11.16 docker image directrly
* **Python version**: 3.10.5 (+ latest 3.9 and 3.8)
* **Platform**: Ubuntu 64 and docker
* **Do you use pgbouncer?**: No
* **Did you install asyncpg with pip?**: Yes
* **If you built asyncpg locally, which version of Cython did you use?**: N/A
* **Can the issue be reproduced under both asyncio and
[uvloop](https://github.com/magicstack/uvloop)?**: I only tried asyncio

We observed PODs running out of memory.
They are reading from a queue constantly and calling this function

```python
async def process_request(message, db : ):
await db.connect()
# do_something(message, db)
await db.disconnect()
```

I am confident this is related to asyncpg as here are some of the objects that are created in memory with every iteration of connect/disconnect.

```
asyncpg.pgproto.pgproto.ReadBuffer | 1000 | 132.81 KB
collections.OrderedDict | 1000 | 125.00 KB
asyncpg.pool.PoolConnectionHolder | 1000 | 117.19 KB
asyncio.events.TimerHandle | 1001 | 109.48 KB
```

I have tried:
- waiting for the TimerHandle to expire (60s)
- calling engine.dispose() manually
- calling the garbage collector manually

Sometime in the past we encountered a problem that made 10000s of connections in a few seconds (crashing the DB) that was fixed by abandoning this connect/disconnect pattern, so maybe these two are related.

The db is initialised as such
```python
databases.Database(db_uri)
```

This problem does not exists in the sqlite driver.

Beitragsleitfaden

Für dieses Repository ist kein Beitragsleitfaden indexiert

Erste Schritte

  1. Lies das ganze Issue und danach den Beitragsleitfaden des Projekts.
  2. Schreib ins Issue, dass du es übernimmst — das erspart doppelte Arbeit.
  3. Forke das Repository und arbeite in einem Branch.
  4. Öffne einen Pull Request, der die Issue-Nummer nennt.

Rechercherichtung

Start by reproducing the repeated db.connect() and db.disconnect() pattern described with asyncpg 0.25.0, PostgreSQL 11.16, Python 3.10.5, and the postgres:11.16 Docker image. Inspect whether asyncpg.pgproto.pgproto.ReadBuffer, OrderedDict, PoolConnectionHolder, and asyncio.events.TimerHandle objects remain after disconnect and garbage collection. Compare the behavior with the sqlite driver and determine whether the retained allocations represent a leak.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Bewertung

Tech-Stack
postgresql, python
Bereich
backend, databases
Issue-Typ
Bug
Schwierigkeit
4/5
Geschätzter Aufwand
3-5 Tage
Aktivitätsstatus
Veraltet
Klarheit
Muss geklärt werden
Anfängerfreundlichkeit
25/100

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.