koala73 / koala73/worldmonitor
feat(cyber): add NVD CVE corpus, Shodan InternetDB, and a network-OSINT lookup layer
- Dominant language
- TypeScript
- Stars
- 86.6k
- Forks
- 13.1k
- Avg merge
- 8h 4m
- Merged PRs (30d)
- 825
Description
## Summary
Our cyber surface covers **threat feeds** (what is being actively exploited / seen in the wild)
but has no **vulnerability corpus** and no **network-OSINT lookup layer**.
Repo-wide grep of current cyber upstreams:
```
155 KEV (CISA Known Exploited Vulnerabilities)
77 AbuseIPDB
62 URLhaus (abuse.ch)
27 CISA
6 ransomware.live
4 otx.alienvault
3 feodotracker (abuse.ch)
0 NVD
0 internetdb.shodan.io
0 BGPView / RIPEstat / crt.sh / RDAP
```
CISA KEV is ~1,300 entries — deliberately only the *known-exploited* subset. We have no view
of the ~300k-entry CVE corpus, and no way to answer "what is exposed at this IP / ASN / domain."
## 1. NVD — full CVE corpus
- `services.nvd.nist.gov/rest/json/cves/2.0` — probed 2026-08-05: **`HTTP 200`**, keyless.
- Public domain (US Federal).
- Keyless is heavily rate-limited; a free API key raises the ceiling substantially. Budget for
the key and for incremental sync via `lastModStartDate`, not full re-pulls.
**Why:** KEV tells us what is being exploited *now*; NVD tells us what exists, with CVSS,
CWE, and affected-product data (CPE). That is what makes CVE mentions in advisories and
security-news items resolvable to severity and affected vendor.
`../shadowbroker` also fans out to two CVE fallbacks worth copying for resilience:
MITRE CVE Services `cveawg.mitre.org/api/cve/{id}` (`osint/lookups.py:338`) and
CIRCL `cve.circl.lu/api/cve/{id}` (`:346`).
## 2. Shodan InternetDB — keyless exposure lookup
- `internetdb.shodan.io/{ip}` — probed 2026-08-05: **`HTTP 200`**, 199 bytes for `1.1.1.1`. **Keyless.**
- Returns open ports, hostnames, CPEs, tags, and known CVEs per IP.
**Why:** I previously wrote Shodan off as paid-only. That was wrong — InternetDB is the free
tier and needs no account. It is the cheapest possible "what is exposed here" primitive.
Wired in `../shadowbroker` at `backend/services/osint/lookups.py:444`.
## 3. Network OSINT lookups
| Source | Endpoint | Probe result (2026-08-05) | shadowbroker |
|---|---|---|---|
| RIPEstat | `stat.ripe.net/data/whois/data.json` | **`200`** · 3.3 KB | `osint_intel/resolve.py:149` |
| RIPEstat AS overview | `stat.ripe.net/data/as-overview/data.json` | **`200`** · 666 B | — |
| RDAP.org | `rdap.org/domain/{d}` | **`200`** (302 → registry, expected; RDAP.org is a bootstrap redirector) | `osint/lookups.py:133` |
| BGPView | `api.bgpview.io/asn/{n}`, `/ip/{ip}`, `/prefixes`, `/peers` | **UNVERIFIED** — did not resolve from this machine (`dig` returned no A record, `http=000`). Could be dead or locally blocked. **Confirm before building on it.** | `osint/lookups.py:291-300` |
| crt.sh | `crt.sh/?q=%25.{domain}&output=json` | **UNVERIFIED** — returned `404` for both a test domain and `google.com`; the JSON endpoint may have changed. **Confirm before building on it.** | `osint/lookups.py:209` |
RIPEstat and RDAP are confirmed good and between them cover WHOIS, ASN, and domain
registration. If BGPView is genuinely gone, RIPEstat's routing endpoints are the natural
substitute for ASN/prefix/peer data — do not treat BGPView as a hard dependency.
**Why:** these turn an IP or domain appearing in an advisory, an outage, or a threat feed into
attributable infrastructure — which ASN, which country, which registrar. That is the join
between our cyber surface and the internet-outage work (IODA / RIPE Atlas) in the new-sources issue.
## Proposed work
1. NVD incremental sync (`lastModStartDate`) into a CVE reference store; register for the free
API key. Fall back to MITRE CVE Services and CIRCL on a per-CVE miss.
2. Enrich existing KEV / advisory / security-news records with CVSS, CWE, and affected CPE from NVD.
3. Add Shodan InternetDB as a keyless per-IP exposure lookup.
4. Add a network-OSINT lookup layer on RIPEstat + RDAP. Verify BGPView and crt.sh first; drop
or substitute whichever does not check out.
5. Rate-limit and cache all of the above — these are courtesy-access public services.
## Acceptance criteria
- [ ] Any CVE ID mentioned in an advisory resolves to CVSS + CWE + affected products.
- [ ] NVD sync is incremental and survives the keyless rate limit without dropping records.
- [ ] Per-IP exposure lookup available and cached.
- [ ] BGPView and crt.sh either confirmed working and wired, or explicitly dropped with the reason recorded.
- [ ] Attribution added for each source (see the attribution page issue).
## Related
- New-sources issue (IODA + RIPE Atlas) — shares the RIPE dependency and the outage/attribution join.
- Attribution page issue.
Contributor guide
Research direction
Start by reviewing the existing cyber upstream integrations, then inspect the referenced shadowbroker entry points: osint/lookups.py:338, :346, :444, osint/lookups.py:133, and osint_intel/resolve.py:149. Verify BGPView and crt.sh before treating them as dependencies, and account for incremental NVD sync, rate limits, caching, and attribution. Done means the listed acceptance criteria are met across CVE enrichment, per-IP exposure lookup, network OSINT, and source attribution.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- backend, networking, security
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100