research: C-vs-PyTorch training-dynamics divergence (bit-parity-invisible)

## Why this issue exists

This finding is real, investigated, and currently documented **only in machine-local, gitignored notes** on one development machine (`docs/superpowers/` is gitignored). This issue preserves the knowledge so it survives beyond that machine. **It is documentation of an open research question — no code change is planned or requested here.**

## Phenomenon

**Inference bit-parity is exact (example-level: HAR int32-identical, ECG float-allclose) and unit gold tests pass within tolerance — yet C and PyTorch, trained with the same init scheme (independent random draws on each side), reach different optima over epochs.** The divergence is invisible to every automated check we have: bit-parity exercises inference only, and the controlled single-step comparison is clean apart from benign, punctual routing events (below).

Evidence (2026-06-26, examples-cleanup work; ECG5000 reconstruction AE, SGD(lr=0.005, mom=0.9), 200 epochs, MSE — same init scheme on both sides):

| init scheme (both sides) | C test_mse / AUC | PyTorch test_mse / AUC |
|---|---|---|
| He √2 | 0.198 / 0.868 | 0.630 / 0.824 (collapsed) |
| PyTorch default | 0.285 / 0.960 | 0.177 / 0.929 |

Trajectories track almost exactly for the first epochs (He √2: C 1.50/1.01/0.97 vs PT 1.55/1.02/0.98), then split. The fragile ~1.5K-param AE amplifies per-step differences into different basins; the robust HAR classifier absorbs the same effect and stays within tolerance. `kws_raw` showed a from-scratch accuracy gap in the same class (C ~0.17 vs PT ~0.59 in the original from-scratch runs).

Matching the init scheme (#250 / Issue C) did **not** close the gap — it only moved which side/metric was off. The ECG demo comparison was made informational (PR #251) to neutralize demo impact; this issue is the underlying question. Note: training both sides from **truly identical initial weights** (the decisive experiment designed in the 2026-06-26 finding) was never built — see Open question.

## What has been ruled out

The observer-trace facility (#257, PR #260) compares C vs PyTorch tensor-by-tensor at every probe (forward activations, act-grads, per-layer param-grads, weight deltas) for one controlled step from identical loaded weights, aggregated over 10 non-overlapping batches (kws_raw, FLOAT32):

- **Forward, act-grad wire, optimizer step: clean to float noise.** Loss |C−PT| ≤ 4e-6 per batch; forward max_abs ≤ 3.5e-4; act-grads mean ≤ 8.6e-6; per-step weight deltas ≤ 5.3e-5.
- **The param-grad tier is NOT noise-level — and that was resolved benign.** Early-layer param-grads (first LayerNorm, `ln1`, mean 3.36e-2 / max 3.36e-1 — full gradient height, orders above float noise) looked like a backward bug. Root cause (2026-07-03, bit-level): a **MaxPool argmax tie under sub-ulp forward noise** — C hits a bit-exact tie (first-index-wins, strict `>`), PyTorch's forward differs by 1 ulp and routes the full window gradient (−0.1287) to a different position; per-element dGamma/dBeta inherit the swap at full height. The C LayerNorm backward is formula-identical to PyTorch (dGamma ≡ dy⊙x̂ to ≤7.1e-8; PT matches biased-var/eps-inside to ≤1.2e-7). The batch-0 signature was 2 samples (28: 99.9%, 25: 0.1%), and the sweep headline number was a metric artifact (mean over batch *maxima*). Such events are **punctual** (isolated ±v pairs); a formula bug would be **area-wide**. **No fix needed; no re-gold cascade.**
- **Old on-disk trace dumps were a patchwork of multiple invocations** (provenance forensics, 2026-07-03) — not usable as baselines. Fresh baselines were re-collected at the post-funnel-migration tip and the headline ln1 localization reproduces bit-for-bit to printed precision (expected: the migration PRs were SYM-scoped; kws_raw is pure FLOAT32).

What remains as the plausible mechanism: **accumulation-order / FMA-level per-step differences plus discrete routing events (MaxPool/ReLU ties under sub-ulp noise) compounding over thousands of steps.** A single tie event moves weights ≤ 5.3e-5 per step — harmless individually; whether event *frequency* separates trajectories is the open question. Related, unverified toolchain angle: the SYM kernels are FMA-contraction-sensitive (see the `-ffp-contract` comments in `src/layer/LayerNorm.c`); whether the gold tolerances absorb a compiler contraction-mode change is flagged in design notes as *unverified*.

## Where the evidence lives (machine-local, gitignored)

On the primary dev machine, under the repo root:

- `docs/superpowers/findings/2026-06-26-c-pytorch-training-divergence.md` — original finding + decisive-experiment design
- `docs/superpowers/results/2026-06-29-kws-raw-trace-findings.md` — single-step localization (recorded printed numbers — the reliable old reference; the raw `.npy` dumps were the patchwork)
- `docs/superpowers/results/2026-07-03-ln1-root-cause-maxpool-tie.md` — ln1 resolution (benign)
- `docs/superpowers/traces/2026-07-03-post-pr1b2/` — fresh trace baselines + README (recipe, provenance note)
- `docs/superpowers/traces/2026-07-03-ln1-wedge/` — ln1 bit-level wedge data
- `.superpowers/sdd/trace-recollect.md`, `.superpowers/sdd/ln1-track-{a-cformula,b-ptmirror,c-empirical}.md` — track evidence

The tooling itself is committed (`TraceApi.h`, `examples/_shared/npy_dump_sink.h`, `examples/_shared/trace_compare.py`, `examples/_shared/trace_sweep.py`; `kws_raw` is the only instrumented example) — but the recipe is documented only in the gitignored notes, so here it is:

```bash
# build
cmake --preset examples && cmake --build --preset examples --target trace_c_kws_raw
# one batch, any size:
./build/examples/examples/kws_raw/trace_c_kws_raw --sample-start 0 --batch 32 --act-samples 4
uv run examples/kws_raw/trace_pytorch.py --sample-start 0 --batch 32 --act-samples 4
uv run examples/_shared/trace_compare.py --example kws_raw --step 0 # localizer table
# N-batch aggregate (≥10× rule):
uv run examples/_shared/trace_sweep.py --batches 10
```

## Open question

Is epoch-scale C-vs-PyTorch training divergence:

- **(a) accepted by design** — per the framework vision (memory-over-accuracy; the float twin is a sanity reference, not a gate), documented as an envelope (forward ≤ 3.5e-4; grad-tier deltas are punctual routing events at full gradient height), or
- **(b) in need of a quantified bound** — via the never-built decisive experiments: train both sides from **identical initial weights** and diff trajectories (the 2026-06-26 design; the StateDict load path exists, only a load-then-train entry point is missing), a **multi-step cross-comparison** (the PyTorch trace side is single-step v1), and/or an **event-frequency study** (how often do tie/routing events fire per epoch; do they separate trajectories?).

Known candidate follow-ups if (b) is chosen (from the 2026-07-03 analysis; **not started**): make `trace_compare`/`trace_sweep` count routing events separately instead of letting them dominate max-abs error; document that C-vs-PT gradient comparisons are only well-defined modulo pool/ReLU tie events; build the multi-step comparison.

Start with docs/superpowers/findings/2026-06-26-c-pytorch-training-divergence.md and the 2026-07-03 root-cause and trace results listed in the issue. Review the committed trace tools and run the documented kws_raw recipe only if needed to understand the evidence. Done requires resolving whether the divergence is an accepted envelope or needs quantified experiments, then preserving that decision in tracked documentation.

Written by the indexing model from the issue text.