aboutcode-org / aboutcode-org/vulnerablecode

Vulnerablecode useragent

Aperta
#410 4 commenti 0 reazioni 0 assegnatari Vedi su GitHub
Data collection feature good first issue
Lingua principale
Python
Stelle
702
Fork
328
Merge medio
3g 8h
PR unite (30g)
3

Descrizione

Much of what we're doing now is scraping the webpages for the required information. Doing such activity without even declaring who we are might not be seen as a very good practice.
We need to settle on a useragent choice. Some websites even block regular scraper useragents. There are two solutions I can think of:

- One consistent and custom useragent
- A list of browser user agents

There could exist benefit of either methods but I'm not so sure.

References:
https://forum.mattermost.org/t/mattermost-website-returning-403-when-headers-contain-the-word-python/11412

Guida per i contributori

Nessuna guida per i contributori indicizzata per questo repository

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.