[ARCH-PROP] Artificial Hippocampus Networks

### Architecture Name

AHN

### Parent issue

#5

### Motivations

Long-context modeling faces a fundamental trade-off between the fidelity of full attention and the efficiency of bounded-memory sequence models. Causal attention preserves exact historical information through a growing KV cache, but its cache grows linearly with context length and its attention cost grows quadratically. RNN-like models instead maintain a fixed-size state with constant per-token update cost, but compress all history and can lose information required for precise long-range recall.

[Artificial Hippocampus Networks](https://arxiv.org/html/2510.07318v2) proposes a hybrid memory mechanism inspired by the multi-store model of memory. A sliding attention window retains recent KV pairs as lossless short-term memory. When a KV pair leaves the window, a learnable recurrent module compresses it into a fixed-size long-term memory state. The current query reads from both the in-window KV cache and the compressed state.

This proposal evaluates whether AHN can retain the long-context quality of full attention more effectively than sliding-window attention under the same lossless-memory budget, while reducing KV-cache growth and long-sequence compute. It also tests whether the architecture remains compatible with the full ArchSpace model lifecycle and preserves short-context capability.

### Proposed Architecture

AHN augments each causal attention layer with a recurrent memory pathway. Let $(k_{t-W}, v_{t-W})$ be the KV pair that just exits a sliding attention window of size $W$. AHN updates its fixed-size memory state as

$$
h_{t-W}=\mathrm{AHN}((k_{t-W}, v_{t-W}), h_{t-W-1}).
$$

The token at position $t$ retains the lossless in-window memory

$$
\{(k_i,v_i)\}_{i=t-W+1}^{t},
$$

and reads from both this window and the compressed history:

$$
y_t=f\left(h_{t-W}, \{(k_i,v_i)\}_{i=t-W+1}^{t}, q_t\right).
$$

For the initial implementation, we propose AHN-GatedDeltaNet (AHN-GDN), the paper's representative instantiation. Per attention head, AHN-GDN consumes the KV pair leaving the window and updates the recurrent state with a gated delta rule. The current query then reads from the compressed state through a gated, query-dependent projection, and this output is added to the sliding-window attention output.

For inputs with length no greater than $W$, AHN remains inactive and the model follows the parent Transformer path. For longer inputs, AHN replaces unbounded historical KV storage with a fixed-size compressed state while preserving standard autoregressive token-by-token decoding.

The initial proposal is limited to AHN-GDN. AHN-Mamba2 and AHN-DeltaNet should be considered only after the AHN-GDN implementation is validated.

### Preliminary Results

- Paper: https://arxiv.org/html/2510.07318v2
- Code: https://github.com/ByteDance-Seed/AHN

### Experiments Plan

All comparisons will use matched parent architecture, data, token budget, context length, initialization, learning-rate schedule, and evaluation intervals. Full-attention, sliding-window, and AHN variants will use the same lossless-memory budget where applicable.