koala73 / koala73/worldmonitor
feat(intel): widen intel-history beyond the first three collector domains
- Dominant language
- TypeScript
- Stars
- 86.6k
- Forks
- 13.1k
- Avg merge
- 8h 4m
- Merged PRs (30d)
- 825
Description
Follow-up to #5694 / #5737.
#5694 said to *"start with 2-3 high-signal domains (conflict, military, energy) before going wide"*. #5737 wired exactly those three. This tracks going wide.
## Candidates
Any seeder publishing timestamped, describable events is a candidate. From the fleet: maritime activity, cyber threats, natural disasters, sanctions, infrastructure/power, health signals, displacement. The wiring cost per collector is small — a record projector plus one `makeSeedHistoryAfterPublish` call — because #5737 factored the hook into a shared factory.
## Do this incrementally, and watch cost per domain
Each added domain multiplies three things: embedding spend at seed time, Convex storage (each row carries a 512-float vector, roughly 4-9KB serialized), and vector-index size, which is the dominant cost. Add domains one or two at a time and check the storage/spend curve against the 180-day retention window before adding more.
Prerequisite: the first three should be accumulating in production, with the growth curve actually measured, before widening. #5736 (ingestion health signal) makes that measurement trustworthy.
## Also open: backfill
Accumulation starts at activation — there is no historical backfill, so the store's depth is bounded by elapsed time. Some sources (ACLED especially) can serve historical windows and could seed real depth immediately. Worth deciding whether that is worth the one-off embedding spend, or whether waiting for organic accumulation is fine.
## Acceptance criteria
- [ ] Per-domain record projectors for the agreed next collectors, wired through the shared factory.
- [ ] Measured storage + embedding cost per domain recorded before and after.
- [ ] An explicit decision on backfill (do it, with a scripted and resumable path — or document why not).
Contributor guide
Research direction
Start by reading the shared makeSeedHistoryAfterPublish factory and the wiring from #5737, then review the ingestion health signal from #5736 before selecting the next collectors. Done means agreed per-domain record projectors are wired, storage and embedding costs are measured before and after, and backfill has either a scripted resumable path or a documented decision not to do it.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- backend, data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100