psf / psf/requests

`connect` timeout is *not* a TCP connect timeout, but a TCP connect + send the whole HTTP request timeout

Open
#5,263 14 comments 3 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
54.3k
Forks
10.4k
Avg merge
16h 43m
Merged PRs (30d)
3

Description

connect timeout is not a TCP connect timeout, but a TCP connect + send the whole HTTP request timeout.

Expected Result

I expect that the documentation is correct about connect timeout:
The advanced documentation says:

The connect timeout is the number of seconds Requests will wait for your client to establish a connection to a remote machine (corresponding to the connect()) call on the socket. It’s a good practice to set connect timeouts to slightly larger than a multiple of 3, which is the default TCP packet retransmission window.

Actual Result

The connect timeout seems to be the TCP connect timeout + sending the whole HTTP request.

Reproduction Steps

dd if=/dev/zero of=zero.100MB bs=100M count=1
import requests

files = {'file': open('zero.100MB', 'rb')}
requests.post('somehost', timeout=(1, 1000), files=files)

(It needs a host accepting the large POST, I couldn't find any public right now...)

Result:

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/urllib3/connectionpool.py", line 672, in urlopen
    chunked=chunked,
  File "/usr/local/lib/python3.5/dist-packages/urllib3/connectionpool.py", line 387, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib/python3.5/http/client.py", line 1106, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib/python3.5/http/client.py", line 1151, in _send_request
    self.endheaders(body)
  File "/usr/lib/python3.5/http/client.py", line 1102, in endheaders
    self._send_output(message_body)
  File "/usr/lib/python3.5/http/client.py", line 936, in _send_output
    self.send(message_body)
  File "/usr/lib/python3.5/http/client.py", line 908, in send
    self.sock.sendall(data)
  File "/usr/lib/python3.5/ssl.py", line 891, in sendall
    v = self.send(data[count:])
  File "/usr/lib/python3.5/ssl.py", line 861, in send
    return self._sslobj.write(data)
  File "/usr/lib/python3.5/ssl.py", line 586, in write
    return self._sslobj.write(data)
socket.timeout: The write operation timed out

System Information

$ python -m requests.help
{
  "chardet": {
    "version": "3.0.4"
  },
  "cryptography": {
    "version": ""
  },
  "idna": {
    "version": "2.8"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.6.8"
  },
  "platform": {
    "release": "4.15.0-64-generic",
    "system": "Linux"
  },
  "pyOpenSSL": {
    "openssl_version": "",
    "version": null
  },
  "requests": {
    "version": "2.22.0"
  },
  "system_ssl": {
    "version": "1010100f"
  },
  "urllib3": {
    "version": "1.25.7"
  },
  "using_pyopenssl": false
}

Analysis

Digging into urllib3:
https://github.com/urllib3/urllib3/blob/37ba61a8b8120cbd866d057eaa3936f4b140dee0/src/urllib3/connectionpool.py#L370-L390

It sets the connect timeout on the connection, then validate https, then sends the full request, and only then sets the read timeout on the connection socket.

That being said I could not find any urllib3 doc explicitly defining the connect timeout as requests does...

Finally:
https://docs.python.org/3.8/library/socket.html#socket.socket.sendall

Changed in version 3.5: The socket timeout is no more reset each time data is sent successfully. The socket timeout is now the maximum total duration to send all data.

So maybe it did work as a TCP connect timeout + write idle timeout before python 3.5 (which is still not just TCP connect timeout, but closer to it), but now it is clearly not.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the advanced timeout documentation and the referenced urllib3 connectionpool.py section around lines 370-390, then compare the behavior with Python's socket.sendall documentation. Reproduce the large POST timeout case using the provided commands and determine whether the documented semantics or the timeout implementation should change. Done means the connect timeout behavior and documentation agree.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api, backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.