test(turbo): delegated steel envelope fails intermittently with an identical RMS
- Dominant language
- Rust
- Stars
- 467
- Forks
- 54
- Avg merge
- 4h 25m
- Merged PRs (30d)
- 310
Description
`cache::turbo_tests::delegated_steel_envelope_matches_cold_only_fused_over_200_steps` failed once inside a full `cargo test --workspace --profile test-fast --features metal,accelerate` run on M1 Ultra on 2026-09-08, then passed on every re-run.
The failure text was `step 0: steel vs cold-only RMS 4.6824e-1 exceeds 5e-3 (cold_offset=8, hot_offset=1)`. It failed at step 0, not after drift accumulated, and 4.68e-1 against a 5e-3 threshold is two orders of magnitude out, which is too large to read as f16 reduction jitter.
It did not reproduce: three standalone runs of that test passed, and a full `-p mlxcel-core` run passed 1691 of 1691 with the same test green. The gate that surfaced it passed 10069 of 10070 overall, and an earlier full gate on the same day passed 10530 of 10530 with this test included.
The tree at the time carried only `src/models/mistral4.rs` and `src/models/ministral3.rs` changes, neither of which is reachable from `mlxcel-core`'s Turbo-KV path, so this is not attributable to that work. Filing it rather than discarding it because this is one of the three in-tree fused Metal kernel launchers that the repository's contributor guide names for re-validation after an MLX pin bump (`turbo4_delegated_sdpa.cpp::turbo4_delegated_steel_sdpa`), so an intermittent envelope mismatch there is the wrong thing to leave unrecorded.
Worth checking whether it is a cross-suite interaction rather than a property of the kernel: lablup/mlxcel#1008 records two concurrent `mlxcel-core` suites aborting on the shared Metal device, and lablup/mlxcel#163 records that the core lib suite needs `--test-threads=1`. If it recurs, capture whether anything else was on the GPU at the time.
Contributor guide
Research direction
Start with cache::turbo_tests::delegated_steel_envelope_matches_cold_only_fused_over_200_steps and the turbo4_delegated_sdpa.cpp::turbo4_delegated_steel_sdpa launcher. Run the named standalone and full workspace cargo test commands, including the Metal and Accelerate features, while checking whether other suites share the GPU. Done means reproducing the mismatch or documenting a cross-suite interaction and identifying the relevant test or launcher cause.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, rust
- Domain
- performance, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100