AnswerDotAI / AnswerDotAI/fastcore
[question] Best way to update user-agent used in net.py/urlretrieve?
- Dominant language
- Jupyter Notebook
- Stars
- 1.1k
- Forks
- 295
- Avg merge
- 1d 6h
- Merged PRs (30d)
- 7
Description
Network requests in my corporate network are filtered.
And if user-agent is flagged as 'too old', request is rejected (error 503)
In net.py, there is a user-agent setup at `Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36`
When calling `path = untar_data(URLs.MNIST_SAMPLE)` for example, I got this 503 error.
By updating `url_default_headers['User-Agent']` with `Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36` (the one from my corporate browser) directly in net.py, I got the MNIST data
Is there a way to update smoothly this user-agent (env varible? or anything else?)
Contributor guide
Research direction
Start by reading net.py, especially the url_default_headers User-Agent setting and the urlretrieve path used by untar_data(URLs.MNIST_SAMPLE). Determine how a caller could update the User-Agent without editing net.py, and define what configuration mechanism and behavior would count as a complete solution.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- networking
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100