koala73 / koala73/worldmonitor
feat(history): country timeline & playback — daily snapshots, history API, tier-gated retention (epic)
- Dominant language
- TypeScript
- Stars
- 86.6k
- Forks
- 13.1k
- Avg merge
- 8h 4m
- Merged PRs (30d)
- 825
Description
## Summary
**Country timeline & historical playback (epic)**: persist compact daily per-country snapshots (CII, resilience, top events) server-side, expose a history API + MCP tool, render a country timeline in the UI, and gate retention depth by tier (free 7d / Pro 90d / API 365d). Answers the analyst question free chaos can't: *"what changed since yesterday/last week?"* — and creates the second natural paywall rung after alerts.
## Why (evidence)
- Direct user demand: #4062 (threat level timeline panel), #3840 (collapsible historical profile for markers), recurring timeline asks in Discussions #94.
- Incumbents monetize exactly this: MarineTraffic Essential (£100/mo) gates playback + 7-day voyage history; FR24 ladders history depth (Silver 90d → Gold 365d); Liveuamap premium includes history. History retention is one of the two gates prosumer users consistently cross (the other is alerts).
- Historical context is also the credibility feature for segment 2 (GSOC/supply-chain) and segment 4 (finance) — trend beats snapshot for any analyst writing a report.
## Current state (verified anchors)
- **CII**: `server/worldmonitor/intelligence/v1/get-risk-scores.ts` writes `risk:scores:sebuf:trend-history:v{CII_FORMULA_VERSION}:*` — but only ~24h depth, used for movement labels. No public history endpoint.
- **Resilience**: `server/worldmonitor/resilience/v1/_shared.ts` `historyKey(cc)` — Redis ZSET, 30-day rolling window, entries tagged `${date}:${formula}` (formula-version tagging pattern to reuse). ⚠️ `appendHistory` is write-path only — readers must never call `buildResilienceScore` (it writes); read ZSETs directly.
- **News/events**: live-window only; **no historical persistence exists** for headlines/events.
- **Client**: `src/components/PlaybackControl.ts` already implements a snapshot slider (IndexedDB `getSnapshotTimestamps()`/`loadSnapshot(idx)`) — dashboard-level, local-only. Reusable UI pattern; the gap is server-side data.
- Entitlements plumbing for a tiered numeric limit is established (`convex/config/productCatalog.ts` `PlanFeatures` → `convex/entitlements.ts` → consumers).
## Proposed design
1. **Snapshot writer** (Railway cron, daily ~00:30 UTC, pattern: `seed-military-cii` service): per country write `{date, cii?, resilience?, topEvents: [{title, link, source, category}] (≤5), keySignals}` to `history:country:{cc}` (ZSET keyed by epoch-day; member tagged with formula versions). Tiny footprint: ~200 countries × ~1 KB/day ≈ 75 MB/yr — trim at max tier depth (365d).
2. **API**: `intelligence/v1/get-country-history?country=XX&days=N` (sebuf proto + OpenAPI in `docs/api/`); `N` clamped to the caller's entitlement depth (`historyDays`: free 7, Pro 90, API/Ent 365). Anonymous = free depth.
3. **MCP**: `get_country_history` cache tool (agents asking "how has X trended this month" is a strong MCP use case; respects same depth clamp via existing auth context).
4. **UI**: sparkline/timeline strip on country panel (CII/resilience trend + event markers); clamp + "Unlock 90 days" upsell at the boundary. Later: feed server snapshots into `PlaybackControl` for map-state playback.
## Implementation plan
**Phase 1 — snapshot writer**
- [ ] `scripts/seed-country-history.mjs` reading existing caches (CII scores, resilience scores, feed digest) — read-only composition, no scorer invocation
- [ ] ZSET write + `ZREMRANGEBYSCORE` trim; `seed-meta:history:country` freshness meta + `zeroIsValid` semantics; wire into `/api/health` key sets
- [ ] Railway cron service (shared vars don't auto-inherit; provision like `seed-military-cii`)
- [ ] Backfill note: resilience has 30d ZSETs to seed from; CII starts from go-live
**Phase 2 — proto + endpoint**
- [ ] Proto + generated stubs (follow `docs/` proto-stub guide), endpoint with per-tier depth clamp, OpenAPI YAML, rate-limit policy entry
- [ ] Contract tests incl. clamping, empty-country, formula-version-change behavior (points carry formula tags; do not splice across versions silently — annotate breaks)
**Phase 3 — entitlements**
- [ ] `historyDays: number` in `PlanFeatures` (7/90/365/365) + entitlement tests (transition, watchdog parity)
**Phase 4 — MCP tool**
- [ ] Register in `api/mcp/registry/cache-tools.ts` with output budget; update server-card/conformance fixtures
**Phase 5 — UI**
- [ ] Country panel timeline strip (render trend + event dots; clicking a dot shows that day's top events with links)
- [ ] Upsell at depth boundary reusing `panel-locked-cta` idiom; i18n via `scripts/sync-locale-keys.mjs`
**Phase 6 — docs**
- [ ] Methodology page section (what a snapshot contains, formula-version caveats); blog post ("country risk, but with a memory")
## Acceptance criteria
- After N days of cron runs, `get-country-history` returns N points for Tier-1 countries; anonymous capped at 7, Pro at 90
- UI timeline renders for a followed country with event click-through; depth upsell appears exactly at the entitlement boundary
- MCP tool passes conformance; health endpoint tracks snapshot freshness (STALE_SEED on cron death)
- No new writer touches scorer write paths (read-only composition verified in review)
## Out of scope
- Full map-state time-slider playback across all 56 layers (follow-up once snapshots exist)
- Per-headline full-text archive (store title/link/source only — ToS-safe)
- Backfilled deep history purchases
## Effort
~2–3 weeks across phases; Phases 1–3 deliver standalone value and can ship first.
Contributor guide
Research direction
This epic spans server snapshots, the history API, entitlements, MCP, UI, and documentation; start by narrowing the work to a phase, such as `scripts/seed-country-history.mjs` and Phases 1–3. Read `get-risk-scores.ts`, `resilience/v1/_shared.ts`, `convex/config/productCatalog.ts`, and `convex/entitlements.ts`, then run the relevant contract or entitlement tests. Done means the selected phase meets its listed acceptance criteria without invoking scorer write paths.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- openapi, redis, typescript
- Domain
- api, backend, cloud, database, documentation, frontend
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100