aboutcode-org / aboutcode-org/vulnerablecode
Vulnerablecode useragent
Offen
Data collection
feature
good first issue
- Vorherrschende Sprache
- Python
- Sterne
- 702
- Forks
- 328
- Ø Merge
- 3 T. 8 Std.
- Gemergte PRs (30 T.)
- 3
Beschreibung
Much of what we're doing now is scraping the webpages for the required information. Doing such activity without even declaring who we are might not be seen as a very good practice.
We need to settle on a useragent choice. Some websites even block regular scraper useragents. There are two solutions I can think of:
- One consistent and custom useragent
- A list of browser user agents
There could exist benefit of either methods but I'm not so sure.
References:
https://forum.mattermost.org/t/mattermost-website-returning-403-when-headers-contain-the-word-python/11412
Beitragsleitfaden
Für dieses Repository ist kein Beitragsleitfaden indexiert
Bewertung
Dieses Issue wurde noch nicht bewertet.