Host header not being set for CONNECT method of HTTPS proxy
Open
Nobody has claimed this yet.
Feature Request
- Dominant language
- Python
- Stars
- 54.3k
- Forks
- 10.4k
- Avg merge
- 16h 43m
- Merged PRs (30d)
- 3
Description
Since http.client only supports HTTP 1.0 for the CONNECT method it's not required to send the Host header. Unfortunately, HTTPS proxies often expect a Host header to be present, and fail to connect in its absence. I thought this might be of some interest here since requests + proxies is not particularly uncommon.
Is this something that could be supported in requests?
Expected Result
The expectation is that requests will include the Host header as part of the CONNECTion process.
Actual Result
The Host header is not included.
Reproduction Steps
import requests
>>> requests.get('https://google.ca', timeout=5, verify=False, proxies={'https': 'localhost:8081'})
Traceback (most recent call last):
File "/home/me/.venvs/requests_venv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 594, in urlopen
self._prepare_proxy(conn)
File "/home/me/.venvs/requests_venv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 805, in _prepare_proxy
conn.connect()
File "/home/me/.venvs/requests_venv/lib/python3.6/site-packages/urllib3/connection.py", line 344, in connect
ssl_context=context)
File "/home/me/.venvs/requests_venv/lib/python3.6/site-packages/urllib3/util/ssl_.py", line 344, in ssl_wrap_socket
return context.wrap_socket(sock, server_hostname=server_hostname)
File "/usr/lib/python3.6/ssl.py", line 407, in wrap_socket
_context=self, _session=session)
File "/usr/lib/python3.6/ssl.py", line 817, in __init__
self.do_handshake()
File "/usr/lib/python3.6/ssl.py", line 1077, in do_handshake
self._sslobj.do_handshake()
File "/usr/lib/python3.6/ssl.py", line 689, in do_handshake
self._sslobj.do_handshake()
socket.timeout: _ssl.c:830: The handshake operation timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/me/.venvs/requests_venv/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
timeout=timeout
File "/home/me/.venvs/requests_venv/lib/python3.6/site-packages/urllib3/connectionpool.py", line 638, in urlopen
_stacktrace=sys.exc_info()[2])
File "/home/me/.venvs/requests_venv/lib/python3.6/site-packages/urllib3/util/retry.py", line 398, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='google.ca', port=443): Max retries exceeded with url: / (Caused by ProxyError('Cannot connect to proxy.', timeout('_ssl.c:830: The handshake operation t
imed out',)))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/me/.venvs/requests_venv/lib/python3.6/site-packages/requests/api.py", line 75, in get
return request('get', url, params=params, **kwargs)
File "/home/me/.venvs/requests_venv/lib/python3.6/site-packages/requests/api.py", line 60, in request
return session.request(method=method, url=url, **kwargs)
File "/home/me/.venvs/requests_venv/lib/python3.6/site-packages/requests/sessions.py", line 533, in request
resp = self.send(prep, **send_kwargs)
File "/home/me/.venvs/requests_venv/lib/python3.6/site-packages/requests/sessions.py", line 646, in send
r = adapter.send(request, **kwargs)
File "/home/me/.venvs/requests_venv/lib/python3.6/site-packages/requests/adapters.py", line 510, in send
raise ProxyError(e, request=request)
requests.exceptions.ProxyError: HTTPSConnectionPool(host='google.ca', port=443): Max retries exceeded with url: / (Caused by ProxyError('Cannot connect to proxy.', timeout('_ssl.c:830: The handshake operation tim
ed out',)))
System Information
$ python -m requests.help
{
"chardet": {
"version": "3.0.4"
},
"cryptography": {
"version": ""
},
"idna": {
"version": "2.7"
},
"implementation": {
"name": "CPython",
"version": "3.6.7"
},
"platform": {
"release": "4.15.0-43-generic",
"system": "Linux"
},
"pyOpenSSL": {
"openssl_version": "",
"version": null
},
"requests": {
"version": "2.21.0"
},
"system_ssl": {
"version": "1000207f"
},
"urllib3": {
"version": "1.24.1"
},
"using_pyopenssl": false
}
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Trace the HTTPS proxy path through urllib3/connectionpool.py and urllib3/connection.py, then inspect requests/adapters.py and the linked CPython http.client implementation. Confirm which layer controls CONNECT headers and add or update coverage for the reported proxy case; done means the CONNECT request includes the expected Host header.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100