aboutcode-org / aboutcode-org/vulnerablecode

Vulnerablecode useragent

Ouverte
#410 4 commentaires 0 réactions 0 personnes assignées Voir sur GitHub
Data collection feature good first issue
Langage dominant
Python
Étoiles
702
Forks
328
Merge moyen
3 j 8 h
PR mergées (30 j)
3

Description

Much of what we're doing now is scraping the webpages for the required information. Doing such activity without even declaring who we are might not be seen as a very good practice.
We need to settle on a useragent choice. Some websites even block regular scraper useragents. There are two solutions I can think of:

- One consistent and custom useragent
- A list of browser user agents

There could exist benefit of either methods but I'm not so sure.

References:
https://forum.mattermost.org/t/mattermost-website-returning-403-when-headers-contain-the-word-python/11412

Guide de contribution

Aucun guide de contribution indexé pour ce dépôt

Évaluation

Cette issue n'a pas encore été évaluée.

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.