aio-libs / aio-libs/aiohttp

Growing memory usage in web_protocol.py

Abierto
#10,671 1 comentario 0 reacciones 0 asignados Ver en GitHub
bug reproducer: missing
Lenguaje dominante
Python
Estrellas
16.5k
Forks
2.4k
Merge medio
17 h 22 min
PR fusionados (30 d)
212

Descripción

### Describe the bug

I've been seeing a memory leak on one of our services, when running `tracemalloc`, one of the patterns I can see is that the memory usage in `web_protocol.py` keeps growing when being compared against the first snapshot. See below.

I'm wondering if it's a known issue that `web_protocol.py` isn't cleaning up ?

```
*** top 10 stats ***
/usr/local/lib/python3.10/site-packages/aiohttp/web_protocol.py:380: size=37.5 KiB (+2760 B), count=521 (+48), average=74 B
/usr/local/lib/python3.10/tracemalloc.py:423: size=1024 B (+1024 B), count=13 (+13), average=79 B
/usr/local/lib/python3.10/tracemalloc.py:560: size=608 B (+608 B), count=5 (+5), average=122 B
/usr/local/lib/python3.10/asyncio/base_events.py:438: size=1487 B (+504 B), count=12 (+3), average=124 B
/usr/local/lib/python3.10/tracemalloc.py:315: size=416 B (+416 B), count=8 (+8), average=52 B
/usr/local/lib/python3.10/site-packages/prometheus_client/multiprocess.py:148: size=2304 B (-336 B), count=12 (-2), average=192 B
/usr/local/lib/python3.10/tracemalloc.py:558: size=280 B (+224 B), count=5 (+4), average=56 B
/usr/local/lib/python3.10/site-packages/aiohttp/web_app.py:569: size=904 B (-168 B), count=2 (-1), average=452 B
/usr/local/lib/python3.10/abc.py:123: size=76.4 KiB (-63 B), count=936 (-1), average=84 B
/app/mtcnn_server/app/profiler.py:9: size=448 B (+32 B), count=2 (+1), average=224 B
*** top 10 stats ***
/usr/local/lib/python3.10/site-packages/aiohttp/web_protocol.py:380: size=38.2 KiB (+3450 B), count=533 (+60), average=73 B
/usr/local/lib/python3.10/tracemalloc.py:423: size=1176 B (+1176 B), count=16 (+16), average=74 B
/usr/local/lib/python3.10/tracemalloc.py:560: size=656 B (+656 B), count=6 (+6), average=109 B
/usr/local/lib/python3.10/tracemalloc.py:315: size=520 B (+520 B), count=10 (+10), average=52 B
/usr/local/lib/python3.10/site-packages/prometheus_client/multiprocess.py:148: size=2304 B (-336 B), count=12 (-2), average=192 B
/usr/local/lib/python3.10/asyncio/base_events.py:438: size=1319 B (+336 B), count=11 (+2), average=120 B
/usr/local/lib/python3.10/tracemalloc.py:558: size=336 B (+280 B), count=6 (+5), average=56 B
/app/mtcnn_server/app/profiler.py:9: size=480 B (+64 B), count=2 (+1), average=240 B
/usr/local/lib/python3.10/abc.py:123: size=76.4 KiB (-63 B), count=936 (-1), average=84 B
:672: size=85.2 KiB (+0 B), count=916 (+0), average=95 B
*** top 10 stats ***
/usr/local/lib/python3.10/site-packages/aiohttp/web_protocol.py:380: size=38.9 KiB (+4140 B), count=545 (+72), average=73 B
/usr/local/lib/python3.10/tracemalloc.py:423: size=1328 B (+1328 B), count=19 (+19), average=70 B
/usr/local/lib/python3.10/tracemalloc.py:560: size=704 B (+704 B), count=7 (+7), average=101 B
/usr/local/lib/python3.10/tracemalloc.py:315: size=624 B (+624 B), count=12 (+12), average=52 B
/usr/local/lib/python3.10/asyncio/base_events.py:438: size=1487 B (+504 B), count=12 (+3), average=124 B
/usr/local/lib/python3.10/site-packages/prometheus_client/multiprocess.py:148: size=2304 B (-336 B), count=12 (-2), average=192 B
/usr/local/lib/python3.10/tracemalloc.py:558: size=392 B (+336 B), count=7 (+6), average=56 B
/usr/local/lib/python3.10/site-packages/aiohttp/web_app.py:569: size=904 B (-168 B), count=2 (-1), average=452 B
/app/mtcnn_server/app/profiler.py:9: size=480 B (+64 B), count=2 (+1), average=240 B
/usr/local/lib/python3.10/abc.py:123: size=76.4 KiB (-63 B), count=936 (-1), average=84 B
*** top 10 stats ***
/usr/local/lib/python3.10/site-packages/aiohttp/web_protocol.py:380: size=39.5 KiB (+4830 B), count=557 (+84), average=73 B
/usr/local/lib/python3.10/tracemalloc.py:423: size=1480 B (+1480 B), count=22 (+22), average=67 B
/usr/local/lib/python3.10/tracemalloc.py:560: size=752 B (+752 B), count=8 (+8), average=94 B
/usr/local/lib/python3.10/tracemalloc.py:315: size=728 B (+728 B), count=14 (+14), average=52 B
/usr/local/lib/python3.10/asyncio/base_events.py:438: size=1487 B (+504 B), count=12 (+3), average=124 B
/usr/local/lib/python3.10/tracemalloc.py:558: size=448 B (+392 B), count=8 (+7), average=56 B
/usr/local/lib/python3.10/site-packages/prometheus_client/multiprocess.py:148: size=2304 B (-336 B), count=12 (-2), average=192 B
/usr/local/lib/python3.10/site-packages/aiohttp/web_app.py:569: size=904 B (-168 B), count=2 (-1), average=452 B
/app/mtcnn_server/app/profiler.py:9: size=480 B (+64 B), count=2 (+1), average=240 B
/usr/local/lib/python3.10/abc.py:123: size=76.4 KiB (-63 B), count=936 (-1), average=84 B
*** top 10 stats ***
/usr/local/lib/python3.10/site-packages/aiohttp/web_protocol.py:380: size=40.2 KiB (+5520 B), count=569 (+96), average=72 B
/usr/local/lib/python3.10/tracemalloc.py:423: size=1632 B (+1632 B), count=25 (+25), average=65 B
/usr/local/lib/python3.10/tracemalloc.py:315: size=832 B (+832 B), count=16 (+16), average=52 B
/usr/local/lib/python3.10/tracemalloc.py:560: size=800 B (+800 B), count=9 (+9), average=89 B
/usr/local/lib/python3.10/asyncio/base_events.py:438: size=1487 B (+504 B), count=12 (+3), average=124 B
/usr/local/lib/python3.10/tracemalloc.py:558: size=504 B (+448 B), count=9 (+8), average=56 B
/usr/local/lib/python3.10/site-packages/prometheus_client/multiprocess.py:148: size=2304 B (-336 B), count=12 (-2), average=192 B
/usr/local/lib/python3.10/site-packages/aiohttp/web_app.py:569: size=904 B (-168 B), count=2 (-1), average=452 B
/app/mtcnn_server/app/profiler.py:9: size=480 B (+64 B), count=2 (+1), average=240 B
/usr/local/lib/python3.10/abc.py:123: size=76.4 KiB (-63 B), count=936 (-1), average=84 B
*** top 10 stats ***
/usr/local/lib/python3.10/site-packages/aiohttp/web_protocol.py:380: size=40.9 KiB (+6210 B), count=581 (+108), average=72 B
/usr/local/lib/python3.10/tracemalloc.py:423: size=1784 B (+1784 B), count=28 (+28), average=64 B
/usr/local/lib/python3.10/tracemalloc.py:315: size=936 B (+936 B), count=18 (+18), average=52 B
/usr/local/lib/python3.10/tracemalloc.py:560: size=848 B (+848 B), count=10 (+10), average=85 B
/usr/local/lib/python3.10/asyncio/base_events.py:438: size=1487 B (+504 B), count=12 (+3), average=124 B
/usr/local/lib/python3.10/tracemalloc.py:558: size=560 B (+504 B), count=10 (+9), average=56 B
/usr/local/lib/python3.10/site-packages/prometheus_client/multiprocess.py:148: size=2304 B (-336 B), count=12 (-2), average=192 B
/usr/local/lib/python3.10/site-packages/aiohttp/web_app.py:569: size=904 B (-168 B), count=2 (-1), average=452 B
/app/mtcnn_server/app/profiler.py:9: size=544 B (+128 B), count=2 (+1), average=272 B
/usr/local/lib/python3.10/abc.py:123: size=76.4 KiB (-63 B), count=936 (-1), average=84 B
*** top 10 stats ***
/usr/local/lib/python3.10/linecache.py:137: size=529 KiB (+529 KiB), count=5311 (+5311), average=102 B
/usr/local/lib/python3.10/site-packages/aiohttp/web_protocol.py:380: size=42.2 KiB (+7590 B), count=605 (+132), average=71 B
/usr/local/lib/python3.10/tracemalloc.py:423: size=1936 B (+1936 B), count=31 (+31), average=62 B
/usr/local/lib/python3.10/sre_compile.py:804: size=3056 B (+1672 B), count=9 (+3), average=340 B
/usr/local/lib/python3.10/tracemalloc.py:535: size=1240 B (+1240 B), count=3 (+3), average=413 B
/usr/local/lib/python3.10/tracemalloc.py:315: size=1040 B (+1040 B), count=20 (+20), average=52 B
/usr/local/lib/python3.10/tracemalloc.py:447: size=896 B (+896 B), count=2 (+2), average=448 B
/usr/local/lib/python3.10/tracemalloc.py:248: size=832 B (+832 B), count=1 (+1), average=832 B
/app/mtcnn_server/app/profiler.py:38: size=832 B (+832 B), count=1 (+1), average=832 B
/usr/local/lib/python3.10/sre_parse.py:199: size=0 B (-832 B), count=0 (-1)
```

The traceback for that line is

```
File "/usr/local/lib/python3.10/site-packages/aiohttp/web_protocol.py", line 380
messages, upgraded, tail = self._request_parser.feed_data(data)
```

### To Reproduce

I think from a simple application like this will show the growing memory usage in `web_protocol.py`. I was just sending images to the `/image` enpdoint.

```
from aiohttp import web
import gc
import asyncio
import tracemalloc
from time import time

import objgraph

import tracemalloc

# list to store memory snapshots
snaps = []

def snapshot():
snaps.append(tracemalloc.take_snapshot())

def display_stats():
stats = snaps[0].statistics('filename')
print("\n*** top 5 stats grouped by filename ***")
for s in stats[:5]:
print(f"\n{str(s)}")

def compare():
first = snaps[0]
for snapshot in snaps[1:]:
stats = snapshot.compare_to(first, 'lineno')
print("\n*** top 10 stats ***")
for s in stats[:10]:
print(f"\n{str(s)}")

def print_trace():
# pick the last saved snapshot, filter noise
snapshot = snaps[-1].filter_traces((
tracemalloc.Filter(False, ""),
tracemalloc.Filter(False, ""),
tracemalloc.Filter(False, ""),
))
largest = snapshot.statistics("traceback")[0]

print(f"\n*** Trace for largest memory block - ({largest.count} blocks, {largest.size/1024} Kb) ***")
for largestt in largest.traceback.format():
print(f"\n{str(largestt)}")

async def hanlder(request):
print(f'read request')
req = await request.json()
return web.Response(text="Request has been receieved")

async def on_startup(app) -> None:
asyncio.create_task(show_memory())

async def show_memory():
print('start tracing memory')

tracemalloc.start(10)
start = tracemalloc.take_snapshot()

snapshot_num = 1
while True:
for _ in range(10):
await asyncio.sleep(2)

gc.collect()
snapshot()

display_stats()
compare()
print_trace()

my_web_app = web.Application()
my_web_app.router.add_route('POST', '/image', hanlder)
my_web_app.on_startup.append(on_startup)
web.run_app(my_web_app)
```

### Expected behavior

That the memory being used in `web_protocol.py` gets cleaned up.

### Logs/tracebacks

```python-traceback
File "/usr/local/lib/python3.10/site-packages/aiohttp/web_protocol.py", line 380
messages, upgraded, tail = self._request_parser.feed_data(data)
```

### Python Version

```console
$ 3.10.12
```

### aiohttp Version

```console
3.11.14
```

### multidict Version

```console
6.1.0
```

### propcache Version

```console
0.2.1
```

### yarl Version

```console
1.18.3
```

### OS

Ubuntu 22.04.5 LTS

### Related component

Server

### Additional context

_No response_

### Code of Conduct

- [x] I agree to follow the aio-libs Code of Conduct

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.