MagicStack / MagicStack/httptools

`on_url` not called intermittently?

Open
#20 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.3k
Forks
107
PR merge metrics
No merged PRs in 30d

Description

From time to time, my server output an error that let me think that Request.on_url is not called for the current stream.

Here is a small script to reproduce the behaviour:

import asyncio

from httptools import HttpRequestParser


class Request:

    def __init__(self):
        self.EOF = False

    def on_url(self, url: bytes):
        self.on_url_called = True

    def on_message_complete(self):
        self.EOF = True


async def serve(reader, writer):
    chunks = 2 ** 16
    req = Request()
    parser = HttpRequestParser(req)
    while True:
        data = await reader.read(chunks)
        parser.feed_data(data)
        if not data or req.EOF:
            break
    assert req.on_url_called
    writer.write(b'HTTP/1.1 200 OK\r\nContent-Length: 2\r\n\r\nOK')
    writer.write_eof()


if __name__ == '__main__':
    loop = asyncio.get_event_loop()
    coro = loop.create_task(asyncio.start_server(serve, '127.0.0.1', 8080))
    server = loop.run_until_complete(coro)
    try:
        loop.run_forever()
    except KeyboardInterrupt:
        print('Bye.')
    finally:
        server.close()
        loop.run_until_complete(server.wait_closed())
        loop.close()

To be run with python file.py

And then benchmark it with something like ab -n 10000 -c 8 'http://127.0.0.1:8080/'

Once every, say, 10000 requests, I get this error:

Task exception was never retrieved
future: <Task finished coro=<serve() done, defined at test_httptools.py:18> exception=AttributeError("'Request' object has no attribute 'on_url_called'",)>
Traceback (most recent call last):
  File "test_httptools.py", line 27, in serve
    assert req.on_url_called
AttributeError: 'Request' object has no attribute 'on_url_called'

Any hint? :)

Thanks!

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the intermittent failure using the script in test_httptools.py and the ab benchmark described in the report. Start at HttpRequestParser and parser.feed_data, then trace how request data reaches Request.on_url. Done means determining why a valid request can complete without setting on_url_called and adding or updating coverage for that behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, networking
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.