psf / psf/requests

A `ConnectionError` ("Read timed out.") is raised instead of `ReadTimeout`, when using `timeout` keyword argument of `Session.get()`

Open
#5,430 11 comments 14 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
54.3k
Forks
10.4k
Avg merge
16h 43m
Merged PRs (30d)
3

Description

Consider the code below (main.py). When a temporary network disconnect occurs without the timeout keyword argument to Session.get(), the client may hang indefinitely and no exception is raised.

However, if I use the timeout keyword argument, the application will raise a ConnectionError from models.py corresponding to the urllib3.exceptions.ReadTimeoutError:

requests.exceptions.ConnectionError: HTTPSConnectionPool(host='confluence.example.net', port=443): Read timed out.

Given that the exception is only raised, when using the timeout keyword argument, why isn't Requests raising ReadTimeout exception instead? In particular, the ConnectionError's exception message "Read timed out." suggests that it should be a ReadTimeout exception?

To mitigate the issue I'm currently performing a regular expression match on the exception message, which is a bad practice:

except ConnectionError as e:
     if re.search('Read timed out', str(e), re.IGNORECASE):

main.py:

 try:
     with requests.Session() as rs:
         rs.mount('https://', HTTPAdapter(max_retries=Retry(total=10, connect=10, read=10, backoff_factor=1)))
         with rs.get(url, params={}, headers={}, auth=self.auth, verify=self.ssl_verify, timeout=(30, 30)) as r:
             r.raise_for_status()
             page_set = r.json()
 except ReadTimeout as e:
     logging.exception('Request for page set timed out: {}'.format(url))
     continue
 except ConnectionError as e:
     if re.search('Read timed out', str(e), re.IGNORECASE):
         logging.exception('Request for page set timed out (network problem): {}'.format(url))
         continue
     else:
         raise

models.py:
https://github.com/psf/requests/blob/master/requests/models.py

def generate():
            # Special case for urllib3.
            if hasattr(self.raw, 'stream'):
                try:
                    for chunk in self.raw.stream(chunk_size, decode_content=True):
                        yield chunk
                except ProtocolError as e:
                    raise ChunkedEncodingError(e)
                except DecodeError as e:
                    raise ContentDecodingError(e)
                except ReadTimeoutError as e:
                    raise ConnectionError(e)

Exception:

ERROR -- 04/19/2020 04:51:32 PM -- root -- ThreadPoolExecutor-0_0  -- Request for page set timed out (network problem): https://confluence.example.net/rest/api/content/search?expand=version,history,space,body.storage,children.attachment.version,children.attachment.history,children.attachment.space&limit=50&start=1900&cql=(type=page)
Traceback (most recent call last):
  File "/Users/nlykkei/projects/atlassian-watchdog/lib/python3.7/site-packages/urllib3/response.py", line 425, in _error_catcher
    yield
  File "/Users/nlykkei/projects/atlassian-watchdog/lib/python3.7/site-packages/urllib3/response.py", line 755, in read_chunked
    chunk = self._handle_chunk(amt)
  File "/Users/nlykkei/projects/atlassian-watchdog/lib/python3.7/site-packages/urllib3/response.py", line 708, in _handle_chunk
    returned_chunk = self._fp._safe_read(self.chunk_left)
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 620, in _safe_read
    chunk = self.fp.read(min(amt, MAXAMOUNT))
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/socket.py", line 589, in readinto
    return self._sock.recv_into(b)
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ssl.py", line 1071, in recv_into
    return self.read(nbytes, buffer)
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ssl.py", line 929, in read
    return self._sslobj.read(len, buffer)
socket.timeout: The read operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/nlykkei/projects/atlassian-watchdog/lib/python3.7/site-packages/requests/models.py", line 751, in generate
    for chunk in self.raw.stream(chunk_size, decode_content=True):
  File "/Users/nlykkei/projects/atlassian-watchdog/lib/python3.7/site-packages/urllib3/response.py", line 560, in stream
    for line in self.read_chunked(amt, decode_content=decode_content):
  File "/Users/nlykkei/projects/atlassian-watchdog/lib/python3.7/site-packages/urllib3/response.py", line 781, in read_chunked
    self._original_response.close()
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/contextlib.py", line 130, in __exit__
    self.gen.throw(type, value, traceback)
  File "/Users/nlykkei/projects/atlassian-watchdog/lib/python3.7/site-packages/urllib3/response.py", line 430, in _error_catcher
    raise ReadTimeoutError(self._pool, None, "Read timed out.")
urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='confluence.example.net', port=443): Read timed out.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/nlykkei/projects/atlassian-watchdog/confluence/confluence.py", line 112, in __producer
    with rs.get(url, params={}, headers={}, auth=self.auth, verify=self.ssl_verify, timeout=(30, 30)) as r:
  File "/Users/nlykkei/projects/atlassian-watchdog/lib/python3.7/site-packages/requests/sessions.py", line 543, in get
    return self.request('GET', url, **kwargs)
  File "/Users/nlykkei/projects/atlassian-watchdog/lib/python3.7/site-packages/requests/sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "/Users/nlykkei/projects/atlassian-watchdog/lib/python3.7/site-packages/requests/sessions.py", line 683, in send
    r.content
  File "/Users/nlykkei/projects/atlassian-watchdog/lib/python3.7/site-packages/requests/models.py", line 829, in content
    self._content = b''.join(self.iter_content(CONTENT_CHUNK_SIZE)) or b''
  File "/Users/nlykkei/projects/atlassian-watchdog/lib/python3.7/site-packages/requests/models.py", line 758, in generate
    raise ConnectionError(e)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='confluence.example.net', port=443): Read timed out.

Expected Result

I would expect a requests.exceptions.ReadTimeout to be raised.

Actual Result

A requests.exceptions.ConnectError was raised instead, with the error message: Read timed out.

System Information

$ python -m requests.help
nlykkei:~$ python3 -m requests.help
{
  "chardet": {
    "version": "3.0.4"
  },
  "cryptography": {
    "version": ""
  },
  "idna": {
    "version": "2.7"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.7.7"
  },
  "platform": {
    "release": "19.4.0",
    "system": "Darwin"
  },
  "pyOpenSSL": {
    "openssl_version": "",
    "version": null
  },
  "requests": {
    "version": "2.23.0"
  },
  "system_ssl": {
    "version": "1010106f"
  },
  "urllib3": {
    "version": "1.24.3"
  },
  "using_pyopenssl": false
}

This command is only available on Requests v2.16.4 and greater. Otherwise,
please provide some basic information about your system (Python version,
operating system, &c).

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in requests/models.py at generate(), where urllib3.exceptions.ReadTimeoutError is converted, and trace how requests/sessions.py accesses response content. Reproduce the reported timeout case with the shown Session.get() call and inspect the existing exception hierarchy. Done means the read-timeout path raises ReadTimeout rather than ConnectionError, with regression coverage for the reported behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
networking
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.