lablup / lablup/mlxcel

test(turbo): delegated steel envelope fails intermittently with an identical RMS

Open
#1,712 2 comments 0 reactions 0 assignees View on GitHub
area:core priority:low status:ready type:test
Dominant language
Rust
Stars
467
Forks
54
Avg merge
4h 25m
Merged PRs (30d)
310

Description

`cache::turbo_tests::delegated_steel_envelope_matches_cold_only_fused_over_200_steps` failed once inside a full `cargo test --workspace --profile test-fast --features metal,accelerate` run on M1 Ultra on 2026-09-08, then passed on every re-run.

The failure text was `step 0: steel vs cold-only RMS 4.6824e-1 exceeds 5e-3 (cold_offset=8, hot_offset=1)`. It failed at step 0, not after drift accumulated, and 4.68e-1 against a 5e-3 threshold is two orders of magnitude out, which is too large to read as f16 reduction jitter.

It did not reproduce: three standalone runs of that test passed, and a full `-p mlxcel-core` run passed 1691 of 1691 with the same test green. The gate that surfaced it passed 10069 of 10070 overall, and an earlier full gate on the same day passed 10530 of 10530 with this test included.

The tree at the time carried only `src/models/mistral4.rs` and `src/models/ministral3.rs` changes, neither of which is reachable from `mlxcel-core`'s Turbo-KV path, so this is not attributable to that work. Filing it rather than discarding it because this is one of the three in-tree fused Metal kernel launchers that the repository's contributor guide names for re-validation after an MLX pin bump (`turbo4_delegated_sdpa.cpp::turbo4_delegated_steel_sdpa`), so an intermittent envelope mismatch there is the wrong thing to leave unrecorded.

Worth checking whether it is a cross-suite interaction rather than a property of the kernel: lablup/mlxcel#1008 records two concurrent `mlxcel-core` suites aborting on the shared Metal device, and lablup/mlxcel#163 records that the core lib suite needs `--test-threads=1`. If it recurs, capture whether anything else was on the GPU at the time.

Contributor guide

Open the contributing guide

Research direction

Start with cache::turbo_tests::delegated_steel_envelope_matches_cold_only_fused_over_200_steps and the turbo4_delegated_sdpa.cpp::turbo4_delegated_steel_sdpa launcher. Run the named standalone and full workspace cargo test commands, including the Metal and Accelerate features, while checking whether other suites share the GPU. Done means reproducing the mismatch or documenting a cross-suite interaction and identifying the relevant test or launcher cause.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, rust
Domain
performance, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.