RFC: Offline memory consolidation ("auto-dream") — idle-time cluster / abstract / supersede

Summary

RFC for an offline memory consolidation capability ("auto-dream"): an opt-in background / idle-time process that clusters related ContextRecords, abstracts them into higher-level memories, links the result to its sources, and retires the redundant raw records — producing more compact, higher-signal online memory over time. Inspired by Generative-Agents "reflection" and the 2025 "sleep-time compute" line.

Storage-native machinery only. The abstraction step is a user-supplied hook; lance-context bundles no models.

Motivation

Long-running agents accumulate large volumes of low-level episodic memory. As the store grows with redundant / near-duplicate entries, recall quality and cost degrade. Consolidation — summarize many raw turns into a few semantic memories, drop redundancy — keeps memory compact and high-signal. lance-context already holds the raw memory and has the primitives to do this in place, versioned and auditable.

Why here (the bet)

Consolidation maps onto primitives lance-context already has:

• embeddings → cluster related / near-duplicate memories
• relationships → link a derived memory back to the source records it summarizes
• lifecycle (supersede / retire) → replace consolidated-away raw records non-destructively (auditable, reversible)
• versioning → each consolidation pass is a new version you can inspect or roll back
• background compaction infra (should_compact, store.rs:1209; start_background_compaction, store.rs:1284) → an existing pattern for scheduling bounded background work

Proposed shape (to discuss)

1. Candidate selection — cluster by embedding similarity, or group by session / run / time window, to find redundant or related sets.
2. Abstraction hook (user-supplied) — given a cluster of records, the user's callback returns a consolidated record (summary / abstraction). lance-context invokes the hook and persists the result; it bundles no models.
3. Write-back — insert derived records (source="consolidated"), link via relationships, supersede / retire the originals, bump a version.
4. Scheduling — opt-in background pass (idle / cadence / threshold-triggered), reusing the compaction scheduler pattern; bounded and cancellable.
5. Safety / reproducibility — never hard-delete (lifecycle only); record consolidation params + source ids; version-pinned and reversible.

Relationship to #96 (important)

Auto-dream and #96 curation are the same core loop — read records → derive new records → supersede originals → version. They differ only in:

• trigger — background / idle (auto-dream) vs. on-demand (curation)
• output — better online memory (auto-dream) vs. trainable export (curation)

This RFC should decide whether to build one shared "derive-and-supersede" engine with two front-ends (consolidation + curation/export) rather than two parallel implementations.

Non-goals

• Not an agent framework: bundles no LLM / prompting; abstraction is a hook.
• No hard deletion — consolidation is lifecycle-based and reversible.
• Not a replacement for compaction (#16): that is physical file optimization; this is semantic consolidation.

Open questions

• Demand — is there a concrete workflow that needs in-store consolidation rather than doing it in the agent layer? (Same validation bar we applied to #96.)
• Clustering approach and thresholds; how to avoid lossy over-abstraction.
• Interaction with search — do consolidated memories rank over raw? do raw records stay searchable, or only the abstraction?
• Shared derive-and-supersede engine with #96 — yes or no?

Acceptance (RFC stage)

• Agreement on scope / non-goals (machinery + hook, no bundled models), the shared-engine decision with #96, and a short design doc under docs/design/ (mirrors how #90 was handled). Implementation tracked in follow-ups.