koala73 / koala73/worldmonitor

feat(intel): widen intel-history beyond the first three collector domains

Open
#5,745 1 comment 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
TypeScript
Stars
86.6k
Forks
13.1k
Avg merge
8h 4m
Merged PRs (30d)
825

Description

Follow-up to #5694 / #5737.

#5694 said to *"start with 2-3 high-signal domains (conflict, military, energy) before going wide"*. #5737 wired exactly those three. This tracks going wide.

## Candidates

Any seeder publishing timestamped, describable events is a candidate. From the fleet: maritime activity, cyber threats, natural disasters, sanctions, infrastructure/power, health signals, displacement. The wiring cost per collector is small — a record projector plus one `makeSeedHistoryAfterPublish` call — because #5737 factored the hook into a shared factory.

## Do this incrementally, and watch cost per domain

Each added domain multiplies three things: embedding spend at seed time, Convex storage (each row carries a 512-float vector, roughly 4-9KB serialized), and vector-index size, which is the dominant cost. Add domains one or two at a time and check the storage/spend curve against the 180-day retention window before adding more.

Prerequisite: the first three should be accumulating in production, with the growth curve actually measured, before widening. #5736 (ingestion health signal) makes that measurement trustworthy.

## Also open: backfill

Accumulation starts at activation — there is no historical backfill, so the store's depth is bounded by elapsed time. Some sources (ACLED especially) can serve historical windows and could seed real depth immediately. Worth deciding whether that is worth the one-off embedding spend, or whether waiting for organic accumulation is fine.

## Acceptance criteria

- [ ] Per-domain record projectors for the agreed next collectors, wired through the shared factory.
- [ ] Measured storage + embedding cost per domain recorded before and after.
- [ ] An explicit decision on backfill (do it, with a scripted and resumable path — or document why not).

Contributor guide

Open the contributing guide

Research direction

Start by reading the shared makeSeedHistoryAfterPublish factory and the wiring from #5737, then review the ingestion health signal from #5736 before selecting the next collectors. Done means agreed per-domain record projectors are wired, storage and embedding costs are measured before and after, and backfill has either a scripted resumable path or a documented decision not to do it.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
backend, data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.