AnswerDotAI / AnswerDotAI/fastcore

[question] Best way to update user-agent used in net.py/urlretrieve?

Open
#513 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
1.1k
Forks
295
Avg merge
1d 6h
Merged PRs (30d)
7

Description

Network requests in my corporate network are filtered.
And if user-agent is flagged as 'too old', request is rejected (error 503)

In net.py, there is a user-agent setup at `Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36`

When calling `path = untar_data(URLs.MNIST_SAMPLE)` for example, I got this 503 error.
By updating `url_default_headers['User-Agent']` with `Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36` (the one from my corporate browser) directly in net.py, I got the MNIST data

Is there a way to update smoothly this user-agent (env varible? or anything else?)

Contributor guide

Open the contributing guide

Research direction

Start by reading net.py, especially the url_default_headers User-Agent setting and the urlretrieve path used by untar_data(URLs.MNIST_SAMPLE). Determine how a caller could update the User-Agent without editing net.py, and define what configuration mechanism and behavior would count as a complete solution.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
networking
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.