windows-build-and-test flakes: two rocm-core tests race on the process-global $ROCM_PATH
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 40
- Forks
- 9
- Avg merge
- 4d 20h
- Merged PRs (30d)
- 59
Description
Summary
The required windows-build-and-test check fails intermittently in rocm-core, on branches that never touched the crate. It is a test-isolation bug, not a Windows bug.
Observed on run 33616465846 (push to main, commit #319):
---- fix::tests::the_path_fix_finds_the_install_the_rest_of_the_cli_found stdout ----
assertion `left == right` failed: fix-6-path must resolve installs the same way examine does
left: ""
Root cause
Two tests in crates/rocm-core/src/fix.rs each drive newest_rocm_install_dir() by setting the process-global $ROCM_PATH, calling, then restoring it:
the_path_fix_finds_the_install_the_rest_of_the_cli_foundplants a real install and expects that path backthe_path_fix_reports_nothing_rather_than_a_directory_with_no_install_in_itplants an empty directory and expects""
Neither takes a lock, and std::env::set_var mutates state shared by every thread in the process. Interleaved, each test reads the other's planted value: the first sees the empty directory and gets "", which is exactly the failure above. The mirror image (the second test getting a non-empty path) is equally possible.
Why only the Windows lane sees it
ci.yml runs the Linux test lanes under cargo nextest run, which uses a process per test — the mutation cannot escape. windows-build-and-test runs cargo test --workspace --all-targets, which runs tests as threads in one shared process. So the race can only ever be observed there. Nothing about the bug is Windows-specific; it is a property of the runner.
The same latent pattern exists in crates/rocm-core/src/lib.rs, where engine_envs_dir_honors_dedicated_root_override mutates ROCM_CLI_ENGINE_ENVS_ROOT the same way.
Reproduction
Under a threaded runner, run the pair repeatedly:
cargo test -p rocm-core --lib fix::tests::the_path_fix -- --test-threads=2
Locally this fails within ~10 iterations, producing the exact mirror of the CI assertion.
Expected behaviour
A test's result does not depend on which other tests happen to run alongside it. windows-build-and-test is a required check, so an intermittent failure there blocks unrelated PRs and trains reviewers to re-run red checks without reading them.
Suggested fix
Pass the override in as an argument instead of through the environment — the crate already has this seam (discover_rocm_installs_in). Where a test genuinely must exercise the env-reading path, the repo already has two process-wide locks for it (ScopedTestEnv and PROCESS_ENV_TEST_LOCK in apps/rocm/); fix.rs was simply the omission. A static contract test can keep the discipline from lapsing again.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with crates/rocm-core/src/fix.rs and run the repeated threaded command for the two path-fix tests. Read discover_rocm_installs_in and the related environment-handling code in crates/rocm-core/src/lib.rs, including the named locks in apps/rocm/. Done means the tests no longer depend on one another's process-global environment and the threaded reproduction remains stable.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- cli, testing-qa
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 72/100