mudler / mudler/parakeet.cpp

test_relpos_attention_local: local (rel_pos_local_attn) attention diverges from NeMo

Open
#44 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
786
Forks
93
PR merge metrics
No merged PRs in 30d

Description

While building a full NeMo-baseline set to run the model-dependent test suite, I
hit a divergence in the local (Longformer) attention path that looks separate
from #39 (the streaming O(N²) fix) — filing it on its own.

Symptom

test_relpos_attention_local fails on the 110m anchor, on CPU (PARAKEET_DEVICE=cpu,
f32 GGUF), so it isn't iGPU fp16 tolerance:

[relpos_attention_local] n=47616 max|d|=3.349e+02 mean|d|=9.779e+00 (worst@47338 got=0.44526 ref=335.37750) -> FAIL

The divergence is broad (mean |d| ≈ 10, not a single element) and the worst point
is the last time frame (worst index 47338 = frame 92 of T=93, d_model=512).

It's not the --att-context-size (W) chosen for the baseline

I regenerated PARAKEET_TEST_BASELINE_LOCAL at two windows and re-ran:

W result
64 worst@47338 got=0.44526 ref=335.37750
32 worst@47338 got=0.44526 ref=356.31686

The C++ output (got) is identical across W while NeMo's ref changes — i.e.
forward_local does not respond to the window the baseline encodes. (W=128 is
correctly rejected by the test since W ≥ T.)

test_relpos_attention_local_chunked and test_relpos_attention_local_memory
pass (they use an internal brute-force reference), so the gap is specific to
the non-chunked forward_local vs the NeMo rel_pos_local_attn baseline.

Reproduce

# baseline (NeMo): local attention with a finite window over speech.wav
python scripts/gen_nemo_baseline.py \
  --model nvidia/parakeet-tdt_ctc-110m \
  --audio tests/fixtures/speech.wav \
  --att-context-size 64 --output /tmp/baseline_local.gguf

# convert the 110m anchor to f32 gguf -> PARAKEET_TEST_GGUF
PARAKEET_DEVICE=cpu \
PARAKEET_TEST_GGUF=/tmp/pk110m-f32.gguf \
PARAKEET_TEST_BASELINE_LOCAL=/tmp/baseline_local.gguf \
  ./build/tests/test_relpos_attention_local

Question

Is this a known limitation, a layout/convention mismatch between the dumped
pos_emb ([2W+1, d_model]) and what forward_local expects, or a real bug in
the non-chunked local path? Happy to dig into forward_local if it's worth a fix.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with forward_local and the test_relpos_attention_local executable, then compare its handling of the dumped pos_emb layout and attention window with scripts/gen_nemo_baseline.py. Reproduce the CPU f32 divergence at W=64 and W=32; done means the C++ output responds to W and matches the NeMo baseline while the chunked and memory tests remain passing.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
machine-learning, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.