tornadoweb / tornadoweb/tornado

SimpleAsyncHTTPClient randomly blocking on "ValueError: IOStream is not idle; cannot convert to SSL"

Open
#2,546 11 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

investigation iostream
Dominant language
Python
Stars
22.2k
Forks
5.6k
Avg merge
3h 42m
Merged PRs (30d)
16

Description

Hi, I'm using SimpleAsyncHTTPClient to perform a few hundred requests per minute. It has been working fine until I started encountering the error below. This error happens randomly (I didn't manage to replicate it) and once it occurs, the SimpleAsyncHTTPClient fails on all his following requests (Timeout while connecting). A process restart solves the problem until the error occurs again. Any idea where this comes from or how to handle it?

I tried catching it and managed to close the client and create a new instance, but it still can't perform HTTP requests until the program is restarted.

ERROR:tornado.application:Exception in callback <functools.partial object at 0x7f6a3894c3c0>
Traceback (most recent call last):
  File "/home/ubuntu/env/local/lib/python2.7/site-packages/tornado/ioloop.py", line 758, in _run_callback
    ret = callback()
  File "/home/ubuntu/env/local/lib/python2.7/site-packages/tornado/stack_context.py", line 300, in null_wrapper
    return fn(*args, **kwargs)
  File "/home/ubuntu/env/local/lib/python2.7/site-packages/tornado/ioloop.py", line 779, in _discard_future_result
    future.result()
  File "/home/ubuntu/env/local/lib/python2.7/site-packages/tornado/concurrent.py", line 261, in result
    raise_exc_info(self._exc_info)
  File "/home/ubuntu/env/local/lib/python2.7/site-packages/tornado/gen.py", line 1141, in run
    yielded = self.gen.throw(*exc_info)
  File "/home/ubuntu/env/local/lib/python2.7/site-packages/tornado/simple_httpclient.py", line 272, in run
    max_buffer_size=self.max_buffer_size)
  File "/home/ubuntu/env/local/lib/python2.7/site-packages/tornado/gen.py", line 1133, in run
    value = future.result()
  File "/home/ubuntu/env/local/lib/python2.7/site-packages/tornado/concurrent.py", line 261, in result
    raise_exc_info(self._exc_info)
  File "/home/ubuntu/env/local/lib/python2.7/site-packages/tornado/gen.py", line 1147, in run
    yielded = self.gen.send(value)
  File "/home/ubuntu/env/local/lib/python2.7/site-packages/tornado/tcpclient.py", line 242, in connect
    server_hostname=host)
  File "/home/ubuntu/env/local/lib/python2.7/site-packages/tornado/iostream.py", line 1368, in start_tls
    raise ValueError("IOStream is not idle; cannot convert to SSL")
ValueError: IOStream is not idle; cannot convert to SSL

I'm using tornado 5.1.1 on Python 2.7.15 and here's a snippet of how I initialize the HTTPClient:

    client = AsyncHTTPClient(force_instance=True)
    client.initialize(max_clients=200)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the traceback entries in tornado/simple_httpclient.py, tornado/tcpclient.py, and tornado/iostream.py, especially start_tls and the AsyncHTTPClient initialization shown in the report. Reproduce the failure under high request concurrency if possible and trace why the IOStream is no longer idle; done means the reported SSL conversion failure no longer leaves subsequent requests timing out, with regression coverage for the failure path.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.