[ARCH-PROP] MARCH

### Architecture Name

March

### Parent issue

#2

### Motivations

Gated DeltaNet compresses the entire causal prefix into a single recurrent state. Although efficient, earlier information may be weakened by decay or overwritten by later updates, and only the latest state remains accessible. MARCH preserves selected earlier states and makes them directly retrievable. We hypothesize that this improves long-context modeling while retaining Gated DeltaNet’s efficiency and short-context quality.

### Proposed Architecture

MARCH retains the original Gated DeltaNet architecture and recurrence. After every (C=512) text tokens, it saves the current cumulative recurrent state as a state anchor without resetting the recurrence.
A learned summary position produces a routing key for each anchor. For every text token, a head-specific routing query scores all causally visible anchors together with a learned null option. The selected anchors are aggregated and read using the token’s original Gated DeltaNet query:
```math
\mathbf{o}_{t,h}
=
\frac{\mathbf{S}_{t,h}\mathbf{q}_{t,h}}{\sqrt{d_k}}
+
\sum_{m \in \mathcal{M}(t)}
\pi_{t,h,m}
\frac{\mathbf{A}^{(m)}_{h}\mathbf{q}_{t,h}}{\sqrt{d_k}}.
```

The current-state branch always remains active, while the null option allows the model to ignore historical states.

### Preliminary Results (if any)

_No response_

### Experiments Plan

## Experiments Plan

| Model size | Baseline | Proposed model |
| --- | --- | --- |
| 1B | OLMo 3 | OLMo 3 + MARCH |
| 3B | OLMo 3 | OLMo 3 + MARCH |
| 7B | OLMo 3 | OLMo 3 + MARCH |

Within each model-size pair, both runs will use the same model dimensions, tokenizer, training data and order, token budget, sequence length, optimizer, learning-rate schedule, batch size, random seed, distributed topology, and evaluation protocol. The only architectural difference will be the addition of MARCH.

Comparisons will include training-loss and downstream evaluation results at matched token counts. Long-context performance, training throughput, peak memory, and inference costs will also be reported.

The proposal will be considered supported if MARCH shows consistent quality improvements across the 1B, 3B, and 7B comparisons without introducing training instability, while its additional computation and memory costs remain practically acceptable.

Start by reading the parent issue #2 and the MARCH proposal here, since no implementation files, tests, or entry points are identified. The stated completion criteria are matched 1B, 3B, and 7B baseline comparisons covering quality, stability, throughput, memory, and inference costs.

Written by the indexing model from the issue text.