koala73 / koala73/worldmonitor

docs(intel): write and drill the embedding-model migration runbook for intelHistory

Open
#5,742 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
87k
Forks
13.2k
Avg merge
8h 4m
Merged PRs (30d)
825

Description

Follow-up to #5694 / #5737 (raised by the data-migration reviewer).

## Problem

The `intelHistory` vector index pins three literals that must agree — `EMBED_DIMS` (`scripts/lib/brief-dedup-consts.mjs`), `INTEL_HISTORY_EMBED_DIMS` (`convex/intelHistory.ts`), and `dimensions:` on the schema's `by_embedding` index. #5737 added a parity test so they cannot silently drift.

What does **not** exist is the procedure for deliberately changing them. Swapping the embedding model or its dimension is not an edit to those numbers: stored vectors were produced under the old model, and mixing vector spaces in one index is worse than a hard failure — search still returns results, ranked against a similarity scale that no longer means anything.

The documented answer is "a versioned table plus a full re-embed", but there is no runbook, no helper, and no drill. The first time we exercise this path will be the first time we exercise it, under whatever pressure motivated the model change.

## Proposed fix

- Write the runbook: new table (or version-suffixed), backfill/re-embed of retained rows, cutover of the query path, retirement of the old table.
- Script the re-embed rather than hand-rolling it under pressure (cost estimate: 180 days of retained rows x one embedding call each, minus cache hits).
- Decide and document whether search degrades to the old index, serves both, or goes cold during cutover.
- Consider persisting the embedding model/version **on each row** so a mixed state is detectable and filterable rather than silent — the cross-model reviewer flagged this as the structural fix.

## Acceptance criteria

- [ ] A runbook in `docs/` covering the full cutover, with the mixed-vector-space hazard stated plainly.
- [ ] A re-embed helper (script or Convex action) that is idempotent and resumable.
- [ ] Either a per-row model/version field, or an explicit written decision not to add one and why.

Contributor guide

Open the contributing guide

Research direction

Start with scripts/lib/brief-dedup-consts.mjs, convex/intelHistory.ts, the schema’s by_embedding index, and the parity test added in #5737. Define the versioned-table cutover, re-embed behavior, and per-row model/version decision, then document and drill the process; done means docs/ contains the runbook and an idempotent, resumable helper exists with an explicit cutover policy.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
ai, databases, documentation
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.