Proxy error when redirect (Caused by ProxyError('Cannot connect to proxy.', timeout('timed out',)))
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 54.3k
- Forks
- 10.4k
- Avg merge
- 16h 43m
- Merged PRs (30d)
- 3
Description
Summary.
When accessing a URL using requests, the redirect causes proxy error because the initial web page DOES require the proxy, but the page it redirects to DOES NOT require the proxy. The program succeeds in using the proxy for the first web page, but when redirecting to the next web page it does not recognize the
Expected Result
After my program executes 'requests.get()', the program should reach the initial website through the proxy, and then redirect to the new website without going through the proxy, and then print the HTTP status code. In short, I expect the program to run and print the HTTP status code (ideally 200).
Actual Result
Instead I get a proxy error when the program tries to access the website it redirects to and the HTTP status code line isn't executed.
I traced this down using python debugger and my findings were as follows...
The proxy and no proxy are set in my environment and these are first set in the program as a dictionary in 'merge_environment_settings', which calls 'get_environ_proxies' which gets the proxy settings from the environment (which is needed at this point so is correct).
NOTE: If the proxy wasn't needed at this point, it would have produced an empty dictionary for 'proxies' as long as the URL domain matched the 'no_proxy' (thanks to the 'should_bypass_proxies' function).
All is fine at this point as the first URL requires the proxy, BUT...
Further down the line when the redirect occurs, within the 'resolve_redirects' function, 'rebuild_proxies' is called. This functions creates a 'new_proxies' variable which copies the original environment proxy settings.
Further along in this function, the 'bypass_proxy' boolean variable is set from the 'should_bypass_proxies' function, this is correctly set to 'True' as the 'no_proxy' matches the new URL (for the redirect website).
At this point, I expect the 'new_proxies' variable to be altered as the bypass proxy has been triggered, but it remains unchanged and the 'rebuild_proxies' function just returns the original proxy settings.
Another note: within 'rebuild_proxies', the below statement is skipped due to 'not bypass_proxy' but in the 'get_environ_proxies' function mentioned earlier is the only place I discovered to actually return an empty dictionary for 'proxies' when it detects a no proxy. So by skipping the below code, it is skipping the only function I discovered to actually remove the proxy settings.
bypass_proxy = should_bypass_proxies(url, no_proxy=no_proxy)
if self.trust_env and not bypass_proxy:
environ_proxies = get_environ_proxies(url, no_proxy=no_proxy)
proxy = environ_proxies.get(scheme, environ_proxies.get('all'))
if proxy:
new_proxies.setdefault(scheme, proxy)
I actually added the following bit of code to the end of 'rebuild_proxies' and it seemed to fix my program, but i think there is an underlying issue worth fixing.
if bypass_proxy:
new_proxies = OrderedDict()
Reproduction Steps
import requests
from requests import Session
def test():
session = Session()
session.verify = '<SSL-cert>'
r = session.get('<url>', timeout=5)
print(r.status_code)
if __name__ == "__main__":
test()
System Information
$ python -m requests.help
{
"chardet": {
"version": "3.0.4"
},
"cryptography": {
"version": "2.4.2"
},
"idna": {
"version": "2.8"
},
"implementation": {
"name": "CPython",
"version": "3.6.5"
},
"platform": {
"release": "4.15.0-29-generic",
"system": "Linux"
},
"pyOpenSSL": {
"openssl_version": "101000af",
"version": "18.0.0"
},
"requests": {
"version": "2.21.0"
},
"system_ssl": {
"version": "1010007f"
},
"urllib3": {
"version": "1.24.1"
},
"using_pyopenssl": true
}
This command is only available on Requests v2.16.4 and greater. Otherwise,
please provide some basic information about your system (Python version,
operating system, &c).
Thanks for reading and hope it can be fixed!
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing merge_environment_settings, get_environ_proxies, resolve_redirects, rebuild_proxies, and should_bypass_proxies using the reported reproduction. Verify how proxy settings are rebuilt when a redirect matches no_proxy, then confirm the request reaches the redirected URL without the proxy and prints its HTTP status code.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100