HarperFast / HarperFast/harper

[Models] Derived caching table for @embed (low-latency writes + model-change backfill)

Open
#750 1 comment 0 reactions 1 assignee Claimed by @heskew View on GitHub
Dominant language
JavaScript
Stars
89
Forks
10
Avg merge
2d 6h
Merged PRs (30d)
200

Description

## Scope

Follow-up to #632 (Phase 5 of #510). #632 shipped `@embed` with **sync-by-default**: the embedder runs during the transaction's `before` phase, so commit blocks on it. This is the right default for the common case (per @kriszyp on #632: "overwhelmingly the mode that should and would be used").

This issue tracks the **derived caching table** pattern for the cases where write latency matters: origin-table writes commit immediately, and a separate derived table holds the vector and follows sync mode against the origin.

> I am more inclined to think the right way to minimize write latency is to move the vector embedding into a derived caching table (that follows the sync mode, but the "origin" table doesn't need to wait for it).
> — @kriszyp on #632

## Why this is meaningful

Sync mode adds an HTTP round-trip (to the embedding backend) to every originating write that touches the source field. For latency-sensitive write paths (high-throughput ingest, user-facing POSTs), that's not acceptable even with a fast local backend. The derived-cache-table pattern keeps origin-write latency at LMDB-speed while still producing the vector in the background — without ad-hoc job infrastructure on top of `server/jobs/`.

## Four things, one primitive

The same iterate-records-and-rewrite primitive (a derived table that embeds by observing origin content changes) lands **four** capabilities #632 deferred:

1. **Low-latency writes** — origin commits without the embedder; derived table embeds and stores asynchronously.
2. **Re-embed-on-model-change backfill** — when `@embed(model: ...)` changes between deploys, existing rows need to be re-embedded through the new model. #632 today only re-INDEXES existing vectors via the schema version-change pathway; it does NOT re-embed. The derived table is the natural place to scan-and-rewrite.
3. **Proactive source-subscribe embedding** — closes a known gap in #632. A caching table sourced from an external source that pushes proactive `put`/`patch` events via `subscribe()` writes them with `isNotification: true`, which `buildEmbedBefore` skips (correctly, for cluster-replication receivers — but the source-push case carries content with no precomputed vector). The skip only bites when the table does not override `get()` (`shouldRevalidateEvents === false`); with a custom `get()`, subscribe events invalidate and the next read re-embeds via `getFromSource`. A derived table that embeds from observed origin content closes this regardless of how the origin was updated (user write, `getFromSource`, or subscribe push).
4. **Write-time-hook timing (default sync mode)** — closes a known limitation in #632. The hook runs in `_writeUpdate` against the in-flight write payload, *before* table validation, so: (a) a write that later fails validation still calls the embedding backend (wasted cost on the reject path), and (b) a tracked-instance mutation (`update(id, {}); row.source = ...; save()`) that sets the source via accessors after `update()` won't re-embed — the source field isn't in that payload. The clean fix — embed at validate-time / the resource layer, against the final committed record — was prototyped and reverted on #632 as a Harper-foreign pattern. Embedding from the *committed, validated* record (which the derived-table / resource-layer approach above does) closes both symptoms: invalid writes never commit (so never observed → no backend call), and the observed record is the final post-merge content.

**Cache-fill failure semantics (note):** if the embedder fails during a `getFromSource` cache-fill, the background cache write aborts *after* the GET has already resolved (the caller already received the source data) — so the row is left uncached and re-embeds on the next read, rather than caching a vectorless row. The derived-table / resource-layer approach should preserve this "don't persist a broken vector" behavior, and is the natural place to add retry/backfill if desired.

## Implementation sketch (to be refined in PR review)

- Component-author flag on `@embed` (or table-level config) opt-in: `@embed(source: ..., model: ..., mode: "derived")` or equivalent. Default stays sync.
- For derived-mode `@embed` attributes:
- Origin write at the put/patch site does NOT chain the embedder into the `before` phase.
- Origin commit triggers an event observed by a paired derived table (Harper resource pattern).
- Derived table's write path runs the embedder in its own `before` phase (i.e. derived-table writes ARE sync against the origin, just deferred).
- Lookup of the vector for query-time follows the derived-table key (origin's PK).
- Model-change backfill: when `attribute.embed.model` differs from `attributeDescriptor.embed.model` at schema load, iterate the derived table and re-embed each row. Sync mode at table load (matches #632's stated default) for small tables; can be lifted to background scan for large.

## Stacks on

- #632 (Phase 5), PR #747 — `@embed` directive, default embedder, write-time hook, auto-HNSW, replication-receiver predicate, model-tracking version. Lands first; this issue stacks on it once it merges.

## Out of scope

- Cross-model migration tooling (`harper kb reembed --from X --to Y`) — separate concern.
- Multimodal embeddings (image, audio).

## Tracking

Part of #510. Follow-up to #632.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.