ROCm / ROCm/rocm-cli

windows-build-and-test flakes: two rocm-core tests race on the process-global $ROCM_PATH

Open
#336 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
40
Forks
9
Avg merge
4d 20h
Merged PRs (30d)
59

Description

Summary

The required windows-build-and-test check fails intermittently in rocm-core, on branches that never touched the crate. It is a test-isolation bug, not a Windows bug.

Observed on run 33616465846 (push to main, commit #319):

---- fix::tests::the_path_fix_finds_the_install_the_rest_of_the_cli_found stdout ----
assertion `left == right` failed: fix-6-path must resolve installs the same way examine does
  left: ""
Root cause

Two tests in crates/rocm-core/src/fix.rs each drive newest_rocm_install_dir() by setting the process-global $ROCM_PATH, calling, then restoring it:

  • the_path_fix_finds_the_install_the_rest_of_the_cli_found plants a real install and expects that path back
  • the_path_fix_reports_nothing_rather_than_a_directory_with_no_install_in_it plants an empty directory and expects ""

Neither takes a lock, and std::env::set_var mutates state shared by every thread in the process. Interleaved, each test reads the other's planted value: the first sees the empty directory and gets "", which is exactly the failure above. The mirror image (the second test getting a non-empty path) is equally possible.

Why only the Windows lane sees it

ci.yml runs the Linux test lanes under cargo nextest run, which uses a process per test — the mutation cannot escape. windows-build-and-test runs cargo test --workspace --all-targets, which runs tests as threads in one shared process. So the race can only ever be observed there. Nothing about the bug is Windows-specific; it is a property of the runner.

The same latent pattern exists in crates/rocm-core/src/lib.rs, where engine_envs_dir_honors_dedicated_root_override mutates ROCM_CLI_ENGINE_ENVS_ROOT the same way.

Reproduction

Under a threaded runner, run the pair repeatedly:

cargo test -p rocm-core --lib fix::tests::the_path_fix -- --test-threads=2

Locally this fails within ~10 iterations, producing the exact mirror of the CI assertion.

Expected behaviour

A test's result does not depend on which other tests happen to run alongside it. windows-build-and-test is a required check, so an intermittent failure there blocks unrelated PRs and trains reviewers to re-run red checks without reading them.

Suggested fix

Pass the override in as an argument instead of through the environment — the crate already has this seam (discover_rocm_installs_in). Where a test genuinely must exercise the env-reading path, the repo already has two process-wide locks for it (ScopedTestEnv and PROCESS_ENV_TEST_LOCK in apps/rocm/); fix.rs was simply the omission. A static contract test can keep the discipline from lapsing again.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with crates/rocm-core/src/fix.rs and run the repeated threaded command for the two path-fix tests. Read discover_rocm_installs_in and the related environment-handling code in crates/rocm-core/src/lib.rs, including the named locks in apps/rocm/. Done means the tests no longer depend on one another's process-global environment and the threaded reproduction remains stable.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
cli, testing-qa
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.