urllib: urlretrieve() seems to ignore provided host header
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 35/100
- Issue type
- Bug
- Clarity
- Needs clarification
- Activity status
- Stale
- Tech stack
- python
- Domain
- networking
Research direction
Start at urllib.request.urlretrieve(), build_opener(), and opener.addheaders, using the supplied Windows/Python 3.10 example to reproduce the Host-header behavior. Compare the request headers with the requests example and determine whether Host is intentionally handled specially; done means the behavior is explained and, if confirmed as a bug, covered by a regression test.
Written by the indexing model from the issue text.
Description
Bug report
I have been having issues confirming this, but urlretrieve seems to ignore the provided Host header even if it's added. It seems to correctly look at User-agent and Referer. I have two functions doing the same download, one with urlretrieve and one with requests. The requests one works as expected and fails in the same way urlretrieve fails if I remove the Host header.
def download_helper(url, fname):
opener = urllib.request.build_opener()
opener.addheaders = [('User-agent', 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:103.0) Gecko/20100101 Firefox/103.0'),
('Referer', "https://www.amd.com/en/support/graphics/amd-radeon-6000-series/amd-radeon-6700-series/amd-radeon-rx-6700-xt"),
('Host' , 'us.download.nvidia.com')]
urllib.request.install_opener(opener)
import ssl
ssl._create_default_https_context = ssl._create_unverified_context
urllib.request.urlretrieve(url, filename=fname)
def download_helper2(url, fname):
my_referer = "https://www.amd.com/en/support/graphics/amd-radeon-6000-series/amd-radeon-6700-series/amd-radeon-rx-6700-xt"
resp = requests.get(url, verify=False, stream=True, headers={
'referer': my_referer,
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:93.0) Gecko/20100101 Firefox/93.0',
'Host' : 'us.download.nvidia.com'
})
total = int(resp.headers.get('content-length', 0))
with open(fname, 'wb') as file:
for data in resp.iter_content(chunk_size=1024):
size = file.write(data)
download_helper2('https://192.229.211.70/Windows/516.94/516.94-desktop-win10-win11-64bit-international-dch-whql.exe', r'516.94-desktop-win10-win11-64bit-international-dch-whql.exe')
Your environment
- CPython versions tested on: 3.10.6
- Operating system and architecture: Windows Server 2022 10.0.20348.887 ; AMD64
- Dominant language
- Python
- Stars
- 77.2k
- Forks
- 36k
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 558
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from python/cpython
-
docs pending
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
stdlib type-feature
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
stdlib type-feature
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
-
build type-bug
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
-
stdlib topic-email type-feature
Difficulty 2/5 1-3 hours Newbie friendliness 70/100
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
qgis/QGIS-Documentation#11275 ·
-
bug priority:normal ready-for-dev
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
OpenHands/extensions#626 · 1 comment ·
-
Difficulty 1/5 Under an hour Newbie friendliness 90/100
CSCfi/sd-search-api#39 ·
-
Difficulty 1/5 Under an hour Newbie friendliness 90/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
StevenBlack/hosts#3255 ·