MagicStack / MagicStack/uvloop

memory from transport buffer not freed after connection_lost

Open
#548 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Cython
Stars
11.9k
Forks
616
PR merge metrics
No merged PRs in 30d

Description

  • uvloop version: 0.17.0
  • Python version: 3.11.4
  • Platform: fedora 38
  • Can you reproduce the bug with PYTHONASYNCIODEBUG in env?: yes, with PYTHONASYNCIODEBUG=1
  • Does uvloop behave differently from vanilla asyncio? How?: yes, keeps resident memory

If I flood transport.write() and then close the connection (e.g. having the client telnet session stop) the memory stays resident.
Under stock asyncio, the memory immediately is released.

To reproduce,

  1. start this program,
  2. watch it in top
  3. , telnet to port 8888,
  4. watch memory usage spike.
  5. close the telnet session,
  6. see that memory usage stays high.
  7. repeat this experiment with stock asyncio runner,
  8. the memory is freed immediately. after telnet session is closed

This test script is a malloc bomb, so make sure you watch memory closely or you will consume all the memory on your machine

#!/usr/bin/env python3


import asyncio
import uvloop


class EchoServerProtocol(asyncio.Protocol):
    def __init__(self, on_connect, on_disconnect):
        self.on_connect = on_connect
        self.on_disconnect = on_disconnect

    def connection_lost(self, transport):
        print('Connection lost')
        self.on_disconnect()
        self.transport = None

    def connection_made(self, transport):
        self.on_connect(transport)
        peername = transport.get_extra_info('peername')
        print('Connection from {}'.format(peername))
        self.transport = transport

    def data_received(self, data):
        message = data.decode()
        print('Data received: {!r}'.format(message))

        print('Send: {!r}'.format(message))
        self.transport.write(data)

        print('Close the client socket')


async def send_data(transport):
    message = b'Hello World!' * 1000
    while True:
        transport.write(message)
        await asyncio.sleep(0)


task = None


def on_connect(transport):
    global task
    task = asyncio.create_task(send_data(transport))


def on_disconnect():
    global task
    task.cancel()


async def start():
    # Get a reference to the event loop as we plan to use
    # low-level APIs.
    loop = asyncio.get_running_loop()

    server = await loop.create_server(
        lambda: EchoServerProtocol(on_connect, on_disconnect),
        '127.0.0.1',
        8888,
    )

    async with server:
        await server.serve_forever()


if __name__ == '__main__':
    # with asyncio.Runner() as runner:
    with asyncio.Runner(loop_factory=uvloop.new_event_loop) as runner:
        runner.run(start())

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by running the supplied Python reproduction with uvloop.new_event_loop, first observing memory after the telnet session closes and then comparing it with the stock asyncio runner. Trace the transport buffer and connection_lost path; done means memory is released after disconnect as it is under asyncio.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.