twisted / twisted/twisted

readBody from twisted.web.client cannot be correctly canceled

Open
#12,499 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug web
Dominant language
Python
Stars
6k
Forks
1.2k
Avg merge
2d 10h
Merged PRs (30d)
10

Description

twisted.internet.defer.AlreadyCalledError exception if you cancel readBody deferred
readBody from twisted.web.client cannot be correctly canceled if connection is hang

How to cause this behavior

To reproduce the problem you need slow server or broken http server which doesn't response full content:

from twisted.web import server, resource
from twisted.web.server import NOT_DONE_YET
from twisted.internet import reactor, defer, endpoints

def printme(request):
    request.write('Message'.encode('utf-8'))
    # there is no request.finish() so the connection hangs


class MySite(resource.Resource):
    isLeaf = True
    
    def render_GET(self, request):
        reactor.callLater(1, printme, request)
        return NOT_DONE_YET


def run():
    endpoints.serverFromString(reactor, "tcp:%s" % 9999).listen(server.Site(MySite()))
    reactor.run()

run()

Client code:

from twisted.web.client import Agent, readBody
from twisted.internet import reactor

def startDownload(response):
    print("response", response.code)

    d = readBody(response)
    reactor.callLater(3, d.cancel)  # cancel should stop hanged connection. From time to time it doesn't close all sockets but it's not the topic of the issue

    d.addBoth(lambda canceled_error: print('Canceled'))


def main():
    a = Agent(reactor)
    d = a.request(b"GET", b"http://127.0.0.1:9999/")
    d.addCallback(startDownload)
    d.addErrback(lambda error: print('Download start error: ', error))

    reactor.callLater(5, lambda: reactor.stop())

if __name__ == "__main__":
    main()
    reactor.run()

Run server and client in separate consoles and you will see exception in client console:

Traceback of exception:

while interacting with body decoder:
Traceback (most recent call last):
  File "/Users/taroved/venv_downloader/lib/python3.11/site-packages/twisted/web/_newclient.py", line 546, in connectionLost
    self.response._bodyDataFinished(
  File "/Users/taroved/venv_downloader/lib/python3.11/site-packages/twisted/web/_newclient.py", line 1057, in dispatcher
    return func(*args, **kwargs)
  File "/Users/taroved/venv_downloader/lib/python3.11/site-packages/twisted/web/_newclient.py", line 1301, in _bodyDataFinished_CONNECTED
    self._bodyProtocol.connectionLost(reason)
  File "/Users/taroved/venv_downloader/lib/python3.11/site-packages/twisted/web/client.py", line 1749, in connectionLost
    self.deferred.errback(reason)
  File "/Users/taroved/venv_downloader/lib/python3.11/site-packages/twisted/internet/defer.py", line 926, in errback
    self._startRunCallbacks(fail)
  File "/Users/taroved/venv_downloader/lib/python3.11/site-packages/twisted/internet/defer.py", line 982, in _startRunCallbacks
    raise AlreadyCalledError
twisted.internet.defer.AlreadyCalledError: 

Correct behavour
I suppose that cancel of hanged client requests should work silently

I wrote dirty fix

What do you think about it?

Testing environment

  • on Linux, Linux 6.8.0-71-generic #71-Ubuntu SMP PREEMPT_DYNAMIC Tue Jul 22 16:52:38 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux DISTRIB_ID=Ubuntu DISTRIB_RELEASE=24.04 DISTRIB_CODENAME=noble DISTRIB_DESCRIPTION="Ubuntu 24.04.2 LTS"
  • on macOS, ProductName: macOS ProductVersion: 15.4.1 BuildVersion: 24E263
  • Twisted version
    • 24.11.0

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the cancellation and body-completion paths shown in twisted/web/client.py and twisted/web/_newclient.py, then reproduce the issue using the slow server and client examples in the report. Trace how readBody handles cancellation before connectionLost runs. Done means canceling the hanging download completes without AlreadyCalledError.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
networking
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.