sequence_section_decoder's x86_64 dispatch gate does not do what its comment says (NEON runs on ARM regardless)
- Dominant language
- Rust
- Stars
- 2
- Forks
- 0
- PR merge metrics
- No merged PRs in 30d
Description
## Summary
`sequence_section_decoder.rs` gates its `incant!` dispatch to `x86_64` with a comment claiming *"NEON autoversion has a known decode correctness issue"* (`4602a83`). Investigated 2026-08-29. **Both halves of that claim are wrong**, and the gate should be kept or removed on purpose rather than on that reasoning. The comment and CLAUDE.md are corrected in `6a54b4d`; this issue tracks the remaining decision.
## 1. The gate does not keep NEON out of the decoder
The `cfg` is only on the call site. The `#[autoversion]` attribute on `fused_decode_execute_fast_inner` is **not** arch-gated, so on aarch64 the `#[cfg(not(...))]` arm calls the autoversion dispatcher, which selects the NEON variant at runtime.
Emitted LLVM IR, aarch64-apple-darwin, `--release`, default features:
```
gated (as shipped): ___arcane_fused_decode_execute_fast_inner_neon -> 8 references
call site un-gated: ___arcane_fused_decode_execute_fast_inner_neon -> 8 references
```
Identical. For contrast, `literals_section_decoder` gates the attribute *and* the call site, and genuinely has no `decode_huffman_stream_neon` on aarch64. The two decoders are not gated alike, and that difference does not look chosen.
So every ARM decode has been running the NEON variant this whole time. If the bug were real it would be live on every ARM machine today, gate or no gate.
## 2. There is no observed NEON decode divergence
Arch-gating the attribute as well, so aarch64 genuinely decodes scalar, and A/B-ing against the shipping build:
* full suite green both ways (271 lib + 6 integration, including `tests::decode_corpus::test_decode_corpus_files` over all 101 `.zst` corpus files, and the 19 conformance tests)
* all 12,842 `examples/byte_identity` frames decoded: **identical** pass/fail label sets (the 458 failures are the separate L16-22 optimal-parser bug in both)
## 3. The original symptom was CRLF, not SIMD
`4602a83` cited "output is truncated by small amounts" on windows-arm and macos-latest. Thirteen minutes later `b2fdf39` recorded that ARM *still* failed with the gate in place — "Root cause unknown — not SIMD related (autoversion already gated to x86_64)" — and skipped the tests. Two commits after that, `7d16d87` ("fixes #1") and `be8af63` found the real cause: git inflating the binary corpus files with CRLF on Windows, so the decoder produced correct LF output and the test compared it against CRLF-inflated expected files. "Truncated by small amounts" is exactly that shape.
## What is left to decide
Removing the `cfg` swaps autoversion's own dispatcher for `incant!`'s token dispatch on ARM. Both are runtime-dispatched to the same NEON variant, so there is no measured gain, and the verification above covered **aarch64-apple-darwin only** — not `windows-11-arm`, which is where the original reports came from. It was therefore left in place.
For reference, the NEON variant that does run is worth about 1.9% on decode:
```
benches/decode_all, aarch64-apple-darwin, criterion 100 samples
aarch64 forced scalar (attribute also gated): 3.8501 ms
shipping (NEON autoversion variant): 3.7762 ms
change: -1.92% [-2.17%, -1.67%], p = 0.00
```
The choice is between (a) un-gating the call site so ARM uses `incant!` like x86 does, and (b) gating the attribute too so ARM really is scalar and the comment becomes true — which costs the 1.9%. Doing nothing is also defensible, now that the code says what it actually does.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.