aboutcode-org / aboutcode-org/vulnerablecode

Scrape Github security advisories using HTML scraping

未关闭
#297 3 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
Data collection
主要语言
Python
星标
702
派生
328
平均合并
3 天 8 小时
30 天内合并 PR
3

描述

The GitHub advisories are somewhat weird:
1. the graphql API data require auth and are incomplete (they do not contain external references)
2. the HTML data at https://github.com/advisories contains more data, BUT this is limited to 40 pages of 25 advisories, meaning only 1000 can be scraped from the browse page when there are about 3019 advisories. The reference are there including quite often the fixing commit

Therefore I think we should use either:
1. a hybrid model where we get the list of advisories from the Graphql API calls and then scrape individual pages
2. a pure HTML model where we issue several searches to browse subset of the data that are less than 40 pages each and hope to hone on the full 3000+ advisories.

Some scraper exists at https://github.com/yusufsn/local-repo/blob/87054815200d3add63f201d9feb1e2bedd18d0d6/code/urls_crawlers.ipynb#L177

贡献指南

这个仓库没有索引到贡献指南

评估

这个 Issue 还没有评估数据。

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。