Flaky tests: cargo test --workspace produces inconsistent failures
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 24.1k
- Forks
- 2k
- Avg merge
- 1d 5h
- Merged PRs (30d)
- 365
Description
When running cargo test --workspace locally on uutils/coreutils, I observed that the set of failing tests changes between runs. This suggests that some tests are flaky or environment-dependent.
Environment
- OS: Ubuntu 22.04
- Rust version: rustc 1.93.0
- uutils/coreutils commit: 6a942ba4039b45c080909b72b68a76bc3929a574
Observed failures
Different runs produce different failures, for example:
Run 1:
- test_df::test_df_arguments_override_themselves
- test_df::test_df_compatible_sync
- test_df::test_df_conflicts_overriding
- test_df::test_df_masked_proc_fallback
Run 2:
- test_df::test_df_masked_proc_fallback
- test_tail::test_follow_truncate_fast
- test_touch::test_touch_changes_time_of_file_in_stdout
Run 3:
- test_df::test_df_masked_proc_fallback
Run 4:
- test_df::test_df_masked_proc_fallback
- test_touch::test_touch_changes_time_of_file_in_stdout
Run 5:
- test_df::test_df_masked_proc_fallback
- test_tr::test_truncate_applies_before_complement_with_class
Expected behavior
Tests should be deterministic and either consistently pass or consistently fail.
Actual behavior
The set of failing tests varies between runs, even without code changes. This makes it difficult to validate local changes before submitting a PR.
Question
If I submit a PR and the CI/CD pipeline fails due to these flaky tests, what is the recommended way to proceed?
Should contributors re-run CI, mark these tests as known flaky, or simply note that failures are unrelated to the PR changes?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing cargo test --workspace on Ubuntu 22.04 with rustc 1.93.0 at commit 6a942ba4039b45c080909b72b68a76bc3929a574. Compare the listed test failures across runs, especially test_df::test_df_masked_proc_fallback and the test_tail, test_touch, and test_tr failures. Done means the flaky or environment-dependent causes are identified and the tests become deterministic or their CI handling is clearly documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100