aboutcode-org / aboutcode-org/vulnerablecode

Vulnerablecode useragent

未關閉
#410 4 則留言 0 個 reaction 已指派 0 人 在 GitHub 檢視
Data collection feature good first issue
主要語言
Python
星號
702
分支
328
平均合併
3 天 8 小時
30 天內合併 PR
3

描述

Much of what we're doing now is scraping the webpages for the required information. Doing such activity without even declaring who we are might not be seen as a very good practice.
We need to settle on a useragent choice. Some websites even block regular scraper useragents. There are two solutions I can think of:

- One consistent and custom useragent
- A list of browser user agents

There could exist benefit of either methods but I'm not so sure.

References:
https://forum.mattermost.org/t/mattermost-website-returning-403-when-headers-contain-the-word-python/11412

貢獻指南

這個儲存庫沒有索引到貢獻指南

評估

這個 Issue 還沒有評估資料。

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。