psf / psf/requests

Possible issue with proxies and TLS versions when using a session.

Open
#6,015 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
54.3k
Forks
10.4k
Avg merge
16h 43m
Merged PRs (30d)
3

Description

Using a session or a request object with the same parameters should yield the same results.

When a proxy is used, and when the target website supports TLS 1.0 and TLS 1.1 (or does not support TLS 1.3, I could not figure it out), a request object works fine, whereas a session throws a SSL Error.

Expected Result

import requests

proxies = {
  'http': 'http://127.0.0.1:8888',
  'https': 'http://127.0.0.1:8888',
}

requests.get('https://sidep.gouv.fr/', proxies=proxies)

session = requests.Session()
session.proxies.update(proxies)
session.get('https://sidep.gouv.fr/')

The two ways to get the data should yield the same result.

Actual Result

The request works, but not with the session:

HTTPSConnectionPool(host='sidep.gouv.fr', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLError(1, '[SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:997)')))
Traceback (most recent call last):
  File "C:\Users\Max\AppData\Local\Programs\Python\Python310\lib\site-packages\urllib3\connectionpool.py", line 696, in urlopen
    self._prepare_proxy(conn)
  File "C:\Users\Max\AppData\Local\Programs\Python\Python310\lib\site-packages\urllib3\connectionpool.py", line 964, in _prepare_proxy
    conn.connect()
  File "C:\Users\Max\AppData\Local\Programs\Python\Python310\lib\site-packages\urllib3\connection.py", line 364, in connect    
    conn = self._connect_tls_proxy(hostname, conn)
  File "C:\Users\Max\AppData\Local\Programs\Python\Python310\lib\site-packages\urllib3\connection.py", line 501, in _connect_tls_proxy
    socket = ssl_wrap_socket(
  File "C:\Users\Max\AppData\Local\Programs\Python\Python310\lib\site-packages\urllib3\util\ssl_.py", line 453, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(sock, context, tls_in_tls)
  File "C:\Users\Max\AppData\Local\Programs\Python\Python310\lib\site-packages\urllib3\util\ssl_.py", line 495, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock)
  File "C:\Users\Max\AppData\Local\Programs\Python\Python310\lib\ssl.py", line 512, in wrap_socket
    return self.sslsocket_class._create(
  File "C:\Users\Max\AppData\Local\Programs\Python\Python310\lib\ssl.py", line 1070, in _create
    self.do_handshake()
  File "C:\Users\Max\AppData\Local\Programs\Python\Python310\lib\ssl.py", line 1341, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:997)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\Max\AppData\Local\Programs\Python\Python310\lib\site-packages\requests\adapters.py", line 439, in send        
    resp = conn.urlopen(
  File "C:\Users\Max\AppData\Local\Programs\Python\Python310\lib\site-packages\urllib3\connectionpool.py", line 755, in urlopen
    retries = retries.increment(
  File "C:\Users\Max\AppData\Local\Programs\Python\Python310\lib\site-packages\urllib3\util\retry.py", line 574, in increment  
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='sidep.gouv.fr', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLError(1, '[SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:997)')))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\Max\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\Max\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "c:\Users\Max\.vscode\extensions\ms-python.python-2021.12.1559732655\pythonFiles\lib\python\debugpy\__main__.py", line 45, in <module>
    cli.main()
  File "c:\Users\Max\.vscode\extensions\ms-python.python-2021.12.1559732655\pythonFiles\lib\python\debugpy/..\debugpy\server\cli.py", line 444, in main
    run()
  File "c:\Users\Max\.vscode\extensions\ms-python.python-2021.12.1559732655\pythonFiles\lib\python\debugpy/..\debugpy\server\cli.py", line 285, in run_file
    runpy.run_path(target_as_str, run_name=compat.force_str("__main__"))
  File "C:\Users\Max\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 269, in run_path
    return _run_module_code(code, init_globals, run_name,
  File "C:\Users\Max\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 96, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "C:\Users\Max\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "c:\Users\Max\testssl.py", line 16, in <module>
    raise e
  File "c:\Users\Max\testssl.py", line 13, in <module>
    session.get('https://sidep.gouv.fr/')
  File "C:\Users\Max\AppData\Local\Programs\Python\Python310\lib\site-packages\requests\sessions.py", line 555, in get
    return self.request('GET', url, **kwargs)
  File "C:\Users\Max\AppData\Local\Programs\Python\Python310\lib\site-packages\requests\sessions.py", line 542, in request     
    resp = self.send(prep, **send_kwargs)
  File "C:\Users\Max\AppData\Local\Programs\Python\Python310\lib\site-packages\requests\sessions.py", line 655, in send        
    r = adapter.send(request, **kwargs)
  File "C:\Users\Max\AppData\Local\Programs\Python\Python310\lib\site-packages\requests\adapters.py", line 514, in send        
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='sidep.gouv.fr', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLError(1, '[SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:997)')))

When using a TLS 1.3 enabled (which seem to mean TLS 1.0 and 1.1 disabled) website, both versions work, for example:

import requests

proxies = {
  'http': 'http://127.0.0.1:8888',
  'https': 'http://127.0.0.1:8888',
}

requests.get('https://example.com/', proxies=proxies)

session = requests.Session()
session.proxies.update(proxies)
session.verify = False
session.get('https://example.com/')

Without the proxy, it works fine for both websites. I spend a couple hours trying with many websites to figure out the breaking cause, and I believe it is the TLS version.

System Information

{
  "chardet": {
    "version": null
  },
  "charset_normalizer": {
    "version": "2.0.9"
  },
  "cryptography": {
    "version": ""
  },
  "idna": {
    "version": "3.3"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.10.1"
  },
  "platform": {
    "release": "10",
    "system": "Windows"
  },
  "pyOpenSSL": {
    "openssl_version": "",
    "version": null
  },
  "requests": {
    "version": "2.26.0"
  },
  "system_ssl": {
    "version": "101010cf"
  },
  "urllib3": {
    "version": "1.26.7"
  },
  "using_charset_normalizer": true,
  "using_pyopenssl": false
}

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the provided session and requests.get reproduction, then trace the referenced paths in requests/sessions.py, requests/adapters.py, urllib3/connectionpool.py, urllib3/connection.py, and urllib3/util/ssl_.py. Compare proxy and TLS handling between the two request paths using the reported Python 3.10, requests 2.26.0, and urllib3 1.26.7 environment; done means the session and direct request behave consistently for the reported proxy and TLS cases.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.