[ARCH-PROP] Dynamic Memory Table

### Architecture Name

Dynamic Memory Table

### Parent issue

#14

### Motivations

Memory tables ([DeepSeek Engram](https://arxiv.org/abs/2601.07372), [LongCat](https://arxiv.org/abs/2601.21204), [L3](https://arxiv.org/abs/2601.21461), etc.) can expand model capacity while efficiently reading only a small subset of parameters per token. This provides a promising way to increase stored knowledge without proportionally increasing active computation.

Existing approaches often determine candidate memories from tokens or local token patterns. Although context may affect how retrieved information is used, the accessible memory itself remains relatively static.

DMT asks whether memory access can adapt more directly to the model’s current semantic state while preserving fast, sparse lookup. We hypothesize that this could improve not only knowledge recall, but also tasks requiring contextualization and composition of retrieved information.

### Proposed Architecture

DMT adds an internal parametric memory table to OLMo 3 with three properties:

- scalable memory capacity;
- context-dependent memory reading;
- bounded sparse access per token.

### Preliminary Results (if any)

Related work supports memory tables as a promising scaling direction:

- [Conditional Memory via Scalable Lookup](https://arxiv.org/abs/2601.07372) studies conditional memory as a complementary sparsity axis.
- [Scaling Embeddings Outperforms Scaling Experts in Language Models](https://arxiv.org/abs/2601.21204) shows that embedding scaling can outperform expert scaling in certain regimes and motivates LongCat-Flash-Lite.
- [L³: Large Lookup Layers](https://arxiv.org/abs/2601.21461) extends efficient lookup tables into decoder layers with context-dependent aggregation.

### Experiments Plan

Compare OLMo 3, a capacity-matched static memory table, DMT, and an active-compute-matched control using the same data and token budget.

Evaluate language-modeling loss, knowledge recall, compositional reasoning, and long-context performance. Ablations will vary memory capacity, access budget, and static versus dynamic reading.

Initial multi-seed experiments will use smaller OLMo 3-compatible models before larger-scale validation. Success requires DMT to consistently outperform both OLMo 3 and the static-memory control, including on tasks beyond local recall, while keeping per-token memory access bounded.