Ideas for improving the state of the PyPA advisory database

I wanted to get a discussion started on how we can improve the overall quality of this database moving forward.

When it was first started, the Google team behind osv.dev built a pipeline to ingest data from the NIST National Vulnerability Database (NVD). This is a very fuzzy process which attempts to extract PyPI package affected entries from things like description, reference URLs etc. If memory serves correctly, most of the logic is within https://github.com/google/osv.dev/tree/master/vulnfeeds. This worked well enough as a starting point, but was dependent on people doing triaging work to clean up the data after the automatic imports. I contributed quite a bit to the initial triaging effort here and eventually became a maintainer.

Then the GitHub Security Advisory database entered and started accepting public contributions. The integrated web form became a very low friction way of submitting improvements to advisories across multiple supported ecosystems, and GitHub's CNA status made it easy for GitHub projects to start publishing security advisories of their own and assigning associated CVEs. This seemed great at the time, but of course it was fully dependent on Microsoft continuing to invest in those efforts, and based on the current state of GHSA data and the growing backlog of community PRs, I suspect that it may now be in decline.

I'll admit, the ease of use and large adoption of the GitHub Security Advisory database meant that I shifted almost all of my triage efforts towards their dataset. My intention was always to keep the data between the two in sync, so that an improvement to the GHSA dataset would feed into the PYSEC dataset, but I never had a chance to build any of that and no one else did either.

So we are now in a place where in many cases the GHSA dataset has more accurate data than we have here, but it is also missing some valid reports that we have here, and due to the reduction in capacity there, it seems difficult to contribute improvements there now.

So I would like to see a renewed effort in getting this dataset into a better state, and I am willing to do some of that work, but I can't do it alone.

I welcome suggestions on how we can accomplish this.

Start by reviewing the advisory database and the OSV vulnfeeds pipeline referenced in the issue, then compare the stated gaps between the GHSA and PYSEC datasets. No specific implementation target is defined; done would require an agreed improvement plan before individual work can be scoped.