[Feature] Pluggable agent memory: a generic backend seam with causal-memory / mem0 as reference implementations
- Dominant language
- Rust
- Stars
- 41k
- Forks
- 3.6k
- Avg merge
- 13h 59m
- Merged PRs (30d)
- 299
Description
## Background
Codewhale's agent memory is currently a hardwired, single-implementation native feature with no pluggable third-party backend entry:
- The `MemoryBackend` enum has only `Native` / `Off` variants (`crates/tui/src/config.rs:1985`) — no `Custom`/`External`/`MCP`.
- `NativeMemoryStore` (`crates/tui/src/native_memory.rs:51`) is a concrete struct (Markdown source + SQLite FTS5 index) with no trait, so a third party cannot supply their own store.
- The plugin activation capability list `PluginActivationCapability` (`crates/tui/src/plugins/activation.rs:26`) has no `Memory` category, so a plugin cannot declare "I provide a memory backend".
- `docs/MEMORY.md` states that native is the only memory system, the planned Moraine MCP backend was removed, and `backend` accepts only `native` / `off`.
Today a third-party memory system can only be side-mounted through the generic MCP channel (most external memory systems expose an MCP server). It can be wired in, but it is not a first-class backend recognized by `[memory]` — `# quick-add`, the `remember` tool, and the `/memory` command all remain pinned to native.
## Goal
Upgrade memory from "single, hardwired implementation" to a **generic, pluggable, optional** seam:
- Keep native as the default, zero-dependency local backend.
- Define a stable backend seam that **any** third-party memory system can implement — not one tailored to a particular vendor.
- Make `[memory] backend` a first-class config choice, with the existing `remember` / recall / `/memory` lifecycle routing to whichever backend is selected; a backend only implements "store" and "retrieve".
- Ship causal-memory and mem0 as the **first two reference implementations** to prove the seam is generic across very different backend shapes — not as the endpoint of the work.
## The seam is open to any memory system
The abstraction is deliberately transport- and vendor-agnostic, so the following (and others) can all be implemented behind the same interface:
- **causal-memory** — causal/decision memory (local, MCP stdio/HTTP).
- **mem0** — fact/preference recall (remote service, API/SDK + MCP).
- **Zep, Letta (MemGPT), LangMem, MemOS, Mem0** — any agent-memory platform exposing a service API or MCP.
- A custom in-house store — implement the trait directly.
The only requirement is that a backend can answer "store an entry" and "retrieve relevant entries for a query".
## Reference implementation #1 (focus): causal-memory (developed in-house)
causal-memory is our open-source agent memory system (Rust, Apache-2.0, `JingxuanC/causal-memory`). Its core differentiator is a **causal core + inhibition modeling**: facts, temporal state, and `decision → outcome` causal edges live on a single SQLite store, driven by a hippocampus-style engine (typed spreading activation with excitatory *and* inhibitory edges, Hebbian co-occurrence reinforcement, Q-value dynamics, immutable SWR consolidation). Agents recall not only *what* happened and *when* it was true, but *why* it worked and *what would happen if* they acted differently.
Why it is the first backend we build:
1. **Causal information is the most fragile type under text compaction.** The causal table lives outside the agent's context window, so compaction cannot touch it — in a real-LLM compaction benchmark causal-memory retains 100% causal-table recall while mem0 drops to 45%. This directly patches the "re-hits the same bug the same wrong way" hole after long-session compaction.
2. **Clearly ahead of mem0 on causal capabilities** (CausalEval): C7 belief update 100% vs 80%, C2 intervention prediction 75% vs 40%, C4 inhibition distinction 80% vs 50%.
3. **Integration surface is ready**: 14 MCP tools (stdio + HTTP transports), a Python `Memory` facade, and an existing DSH plugin shape — wiring into Codewhale's MCP/plugin channel is nearly zero new code.
4. The honest caveat: pure fact recall (LoCoMo 79.1% vs mem0 91.6%) is not its strong suit — that is exactly mem0's strength. The two are complementary, which is why a generic seam beats "pick one".
## Reference implementation #2: mem0
mem0 is the fact-recall specialist (LoCoMo 91.6%) and is chosen as the second reference implementation precisely because its shape is the opposite of causal-memory's: a **remote hosted service** with an API/SDK, rather than a local store. If the seam can absorb both, it is general enough for the rest of the ecosystem.
## Plan (phased)
- **Phase 0 — baseline (already landed)**: native as the only backend, `[memory]` opt-in, `#` quick-add, `remember`/recall tools. Landed in v0.9.x.
- **Phase 1 — define the generic backend seam (core change)**: introduce a `MemoryBackend` trait with a minimal, transport-agnostic method set (`store` / `search` / `prompt_block`, plus a health/`status` probe); add an `External`/`Custom` variant to the `MemoryBackend` enum; make `[memory] backend` resolve a backend by **kind + connection parameters** (process/stdio command, HTTP service URL, or MCP server name) rather than hardcoding vendor names. Native becomes a default impl of the trait; `Off` stays as-is. Add a `Memory` capability to `PluginActivationCapability` so a backend can ship as a plugin bundle.
- **Phase 2 — two reference implementations to prove genericity**: land **causal-memory** first (MCP stdio + HTTP), then **mem0** (remote API/SDK). Success criterion: the trait needed no vendor-specific escape hatches to absorb two very different backend shapes.
- **Phase 3 — bring-your-own-backend**: document the driver/adaptor contract so any third party (Zep, Letta, LangMem, a custom store, …) can add a backend — ideally as an external plugin bundle, without a core change.
- **Phase 4 — wrap-up**: update `docs/MEMORY.md`, `config.example.toml`, and the `[memory]` config contract; add backend selection + fallback (degrade to off or native when a backend is unavailable); document the KV-cache effect (the boundary of memory injected into the system-prompt prefix is unchanged).
## Integration surface
- **Reuse the existing MCP/plugin channel**: causal-memory is already an MCP server and can land first as a plugin bundle (`McpStdio`/`McpRemote`) as the Phase 2 bootstrap — no need to wait for the trait abstraction before trying it out. Any MCP-speaking memory system gets the same path for free.
- **Add a native trait backend**: make `[memory] backend = "causal-memory"` (or any registered backend) a first-class config option, with the `remember` tool, `/memory` command, and `#` quick-add all routing to the selected backend.
- **Open the plugin capability**: a `Memory` activation capability lets a third party ship a backend as a reviewed plugin, exactly like they ship skills, MCP servers, or hooks today.
## References
- Current memory implementation: `crates/tui/src/native_memory.rs`, `crates/tui/src/config.rs` (`MemoryBackend` / `MemoryConfig`)
- Memory docs: `docs/MEMORY.md`
- Plugin activation capabilities: `crates/tui/src/plugins/activation.rs`
- causal-memory: https://github.com/JingxuanC/causal-memory (README includes the CausalEval / compaction-survival benchmarks)
- Original memory EPIC: #489
Contributor guide
Research direction
Start by reading crates/tui/src/native_memory.rs and the MemoryBackend/MemoryConfig definitions in crates/tui/src/config.rs to map the current native-only flow. Then inspect crates/tui/src/plugins/activation.rs and docs/MEMORY.md for plugin and configuration constraints. Done means a generic backend seam supports selection and routes remember, recall, /memory, and # quick-add through the chosen backend, with the documented fallback and reference integrations covered.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust, sqlite
- Domain
- backend, cli, developer-experience
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 32/100