aboutcode-org / aboutcode-org/vulnerablecode
Find a scalable way to archive reference URLS
- 主要语言
- Python
- 星标
- 702
- 派生
- 328
- 平均合并
- 3 天 8 小时
- 30 天内合并 PR
- 3
描述
As originally posted by @ziadhany in this issue, we now have a way to collect reference URLs, but there are issue with doing it at scale, including rate limiting:
- https://github.com/aboutcode-org/vulnerablecode/issues/1724#issuecomment-4110565625
> I think we should use web.archive.org to store archive URLs via their "Save Page Now" endpoint `https://web.archive.org/save/`
>
> https://github.com/akamhy/waybackpy/blob/master/waybackpy/save_api.py.
> Since our URLs are stored in AdvisoryReference, we could add a new field `archive_url` to that model.
>
> One major constraint is the rate limit of 15 requests per minute. With 250,000 URLs to process, the initial run would take about 11.5 days:
>
> 250,000 / (15 * 60 * 24) = 11.57 days
@ziadhany is reaching out to archive.org to find better and improved ways to archive at scale.
贡献指南
这个仓库没有索引到贡献指南
评估
这个 Issue 还没有评估数据。