koala73 / koala73/worldmonitor
docs(intel): write and drill the embedding-model migration runbook for intelHistory
- Dominant language
- TypeScript
- Stars
- 87k
- Forks
- 13.2k
- Avg merge
- 8h 4m
- Merged PRs (30d)
- 825
Description
Follow-up to #5694 / #5737 (raised by the data-migration reviewer).
## Problem
The `intelHistory` vector index pins three literals that must agree — `EMBED_DIMS` (`scripts/lib/brief-dedup-consts.mjs`), `INTEL_HISTORY_EMBED_DIMS` (`convex/intelHistory.ts`), and `dimensions:` on the schema's `by_embedding` index. #5737 added a parity test so they cannot silently drift.
What does **not** exist is the procedure for deliberately changing them. Swapping the embedding model or its dimension is not an edit to those numbers: stored vectors were produced under the old model, and mixing vector spaces in one index is worse than a hard failure — search still returns results, ranked against a similarity scale that no longer means anything.
The documented answer is "a versioned table plus a full re-embed", but there is no runbook, no helper, and no drill. The first time we exercise this path will be the first time we exercise it, under whatever pressure motivated the model change.
## Proposed fix
- Write the runbook: new table (or version-suffixed), backfill/re-embed of retained rows, cutover of the query path, retirement of the old table.
- Script the re-embed rather than hand-rolling it under pressure (cost estimate: 180 days of retained rows x one embedding call each, minus cache hits).
- Decide and document whether search degrades to the old index, serves both, or goes cold during cutover.
- Consider persisting the embedding model/version **on each row** so a mixed state is detectable and filterable rather than silent — the cross-model reviewer flagged this as the structural fix.
## Acceptance criteria
- [ ] A runbook in `docs/` covering the full cutover, with the mixed-vector-space hazard stated plainly.
- [ ] A re-embed helper (script or Convex action) that is idempotent and resumable.
- [ ] Either a per-row model/version field, or an explicit written decision not to add one and why.
Contributor guide
Research direction
Start with scripts/lib/brief-dedup-consts.mjs, convex/intelHistory.ts, the schema’s by_embedding index, and the parity test added in #5737. Define the versioned-table cutover, re-embed behavior, and per-row model/version decision, then document and drill the process; done means docs/ contains the runbook and an idempotent, resumable helper exists with an explicit cutover policy.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- ai, databases, documentation
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100