Hmbown / Hmbown/Codewhale

[Feature] Pluggable agent memory: a generic backend seam with causal-memory / mem0 as reference implementations

Open
#6,050 3 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
41k
Forks
3.6k
Avg merge
13h 59m
Merged PRs (30d)
299

Description

## Background

Codewhale's agent memory is currently a hardwired, single-implementation native feature with no pluggable third-party backend entry:

- The `MemoryBackend` enum has only `Native` / `Off` variants (`crates/tui/src/config.rs:1985`) — no `Custom`/`External`/`MCP`.
- `NativeMemoryStore` (`crates/tui/src/native_memory.rs:51`) is a concrete struct (Markdown source + SQLite FTS5 index) with no trait, so a third party cannot supply their own store.
- The plugin activation capability list `PluginActivationCapability` (`crates/tui/src/plugins/activation.rs:26`) has no `Memory` category, so a plugin cannot declare "I provide a memory backend".
- `docs/MEMORY.md` states that native is the only memory system, the planned Moraine MCP backend was removed, and `backend` accepts only `native` / `off`.

Today a third-party memory system can only be side-mounted through the generic MCP channel (most external memory systems expose an MCP server). It can be wired in, but it is not a first-class backend recognized by `[memory]` — `# quick-add`, the `remember` tool, and the `/memory` command all remain pinned to native.

## Goal

Upgrade memory from "single, hardwired implementation" to a **generic, pluggable, optional** seam:

- Keep native as the default, zero-dependency local backend.
- Define a stable backend seam that **any** third-party memory system can implement — not one tailored to a particular vendor.
- Make `[memory] backend` a first-class config choice, with the existing `remember` / recall / `/memory` lifecycle routing to whichever backend is selected; a backend only implements "store" and "retrieve".
- Ship causal-memory and mem0 as the **first two reference implementations** to prove the seam is generic across very different backend shapes — not as the endpoint of the work.

## The seam is open to any memory system

The abstraction is deliberately transport- and vendor-agnostic, so the following (and others) can all be implemented behind the same interface:

- **causal-memory** — causal/decision memory (local, MCP stdio/HTTP).
- **mem0** — fact/preference recall (remote service, API/SDK + MCP).
- **Zep, Letta (MemGPT), LangMem, MemOS, Mem0** — any agent-memory platform exposing a service API or MCP.
- A custom in-house store — implement the trait directly.

The only requirement is that a backend can answer "store an entry" and "retrieve relevant entries for a query".

## Reference implementation #1 (focus): causal-memory (developed in-house)

causal-memory is our open-source agent memory system (Rust, Apache-2.0, `JingxuanC/causal-memory`). Its core differentiator is a **causal core + inhibition modeling**: facts, temporal state, and `decision → outcome` causal edges live on a single SQLite store, driven by a hippocampus-style engine (typed spreading activation with excitatory *and* inhibitory edges, Hebbian co-occurrence reinforcement, Q-value dynamics, immutable SWR consolidation). Agents recall not only *what* happened and *when* it was true, but *why* it worked and *what would happen if* they acted differently.

Why it is the first backend we build:

1. **Causal information is the most fragile type under text compaction.** The causal table lives outside the agent's context window, so compaction cannot touch it — in a real-LLM compaction benchmark causal-memory retains 100% causal-table recall while mem0 drops to 45%. This directly patches the "re-hits the same bug the same wrong way" hole after long-session compaction.
2. **Clearly ahead of mem0 on causal capabilities** (CausalEval): C7 belief update 100% vs 80%, C2 intervention prediction 75% vs 40%, C4 inhibition distinction 80% vs 50%.
3. **Integration surface is ready**: 14 MCP tools (stdio + HTTP transports), a Python `Memory` facade, and an existing DSH plugin shape — wiring into Codewhale's MCP/plugin channel is nearly zero new code.
4. The honest caveat: pure fact recall (LoCoMo 79.1% vs mem0 91.6%) is not its strong suit — that is exactly mem0's strength. The two are complementary, which is why a generic seam beats "pick one".

## Reference implementation #2: mem0

mem0 is the fact-recall specialist (LoCoMo 91.6%) and is chosen as the second reference implementation precisely because its shape is the opposite of causal-memory's: a **remote hosted service** with an API/SDK, rather than a local store. If the seam can absorb both, it is general enough for the rest of the ecosystem.

## Plan (phased)

- **Phase 0 — baseline (already landed)**: native as the only backend, `[memory]` opt-in, `#` quick-add, `remember`/recall tools. Landed in v0.9.x.
- **Phase 1 — define the generic backend seam (core change)**: introduce a `MemoryBackend` trait with a minimal, transport-agnostic method set (`store` / `search` / `prompt_block`, plus a health/`status` probe); add an `External`/`Custom` variant to the `MemoryBackend` enum; make `[memory] backend` resolve a backend by **kind + connection parameters** (process/stdio command, HTTP service URL, or MCP server name) rather than hardcoding vendor names. Native becomes a default impl of the trait; `Off` stays as-is. Add a `Memory` capability to `PluginActivationCapability` so a backend can ship as a plugin bundle.
- **Phase 2 — two reference implementations to prove genericity**: land **causal-memory** first (MCP stdio + HTTP), then **mem0** (remote API/SDK). Success criterion: the trait needed no vendor-specific escape hatches to absorb two very different backend shapes.
- **Phase 3 — bring-your-own-backend**: document the driver/adaptor contract so any third party (Zep, Letta, LangMem, a custom store, …) can add a backend — ideally as an external plugin bundle, without a core change.
- **Phase 4 — wrap-up**: update `docs/MEMORY.md`, `config.example.toml`, and the `[memory]` config contract; add backend selection + fallback (degrade to off or native when a backend is unavailable); document the KV-cache effect (the boundary of memory injected into the system-prompt prefix is unchanged).

## Integration surface

- **Reuse the existing MCP/plugin channel**: causal-memory is already an MCP server and can land first as a plugin bundle (`McpStdio`/`McpRemote`) as the Phase 2 bootstrap — no need to wait for the trait abstraction before trying it out. Any MCP-speaking memory system gets the same path for free.
- **Add a native trait backend**: make `[memory] backend = "causal-memory"` (or any registered backend) a first-class config option, with the `remember` tool, `/memory` command, and `#` quick-add all routing to the selected backend.
- **Open the plugin capability**: a `Memory` activation capability lets a third party ship a backend as a reviewed plugin, exactly like they ship skills, MCP servers, or hooks today.

## References

- Current memory implementation: `crates/tui/src/native_memory.rs`, `crates/tui/src/config.rs` (`MemoryBackend` / `MemoryConfig`)
- Memory docs: `docs/MEMORY.md`
- Plugin activation capabilities: `crates/tui/src/plugins/activation.rs`
- causal-memory: https://github.com/JingxuanC/causal-memory (README includes the CausalEval / compaction-survival benchmarks)
- Original memory EPIC: #489

Contributor guide

Open the contributing guide

Research direction

Start by reading crates/tui/src/native_memory.rs and the MemoryBackend/MemoryConfig definitions in crates/tui/src/config.rs to map the current native-only flow. Then inspect crates/tui/src/plugins/activation.rs and docs/MEMORY.md for plugin and configuration constraints. Done means a generic backend seam supports selection and routes remember, recall, /memory, and # quick-add through the chosen backend, with the documented fallback and reference integrations covered.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, sqlite
Domain
backend, cli, developer-experience
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.