aboutcode-org / aboutcode-org/vulnerablecode
[Improvement] Standardize 'User-Agent' headers across Importers to prevent blocking
- 主要語言
- Python
- 星號
- 702
- 分支
- 328
- 平均合併
- 3 天 8 小時
- 30 天內合併 PR
- 3
描述
### Description
I have been analyzing the `vulnerablecode/importers` module to understand how data is fetched.
I noticed that several importers make HTTP requests using the default library User-Agents (e.g., `requests` or `aiohttp` defaults).
### The Problem
Using default User-Agents often triggers bot-protection mechanisms on upstream sources (like GitHub, GitLab, or corporate security feeds). This leads to:
1. **403 Forbidden errors** (blocking the importer).
2. **Inconsistent identification** of our traffic to upstream maintainers (good citizenship).
### Proposed Solution
I propose standardizing the identity of the scraper across the codebase.
1. **Define a Constant:** Add a standard `USER_AGENT` string in `vulnerablecode/settings.py` (e.g., `VulnerableCode/1.0 (+https://github.com/aboutcode-org/vulnerablecode)`).
2. **Update Importers:** Audit the `importers/` directory and ensure that `requests.get()` or `ClientSession()` calls explicitly include this header.
### Files Involved
- `vulnerablecode/settings.py`
- `vulnerablecode/importers/*.py` (specifically searching for network calls)
### Request
Could you please assign this issue to me? I can perform the audit and standardize the headers to improve the reliability of the scrapers.
貢獻指南
這個儲存庫沒有索引到貢獻指南
評估
這個 Issue 還沒有評估資料。