tests: test_pp2_create_cudagraphs_last_stage_noop hangs intermittently
- Dominant language
- Python
- Stars
- 17.9k
- Forks
- 4.5k
- Avg merge
- 4d 6h
- Merged PRs (30d)
- 271
Description
## Description
`tests/unit_tests/transformer/test_vision_cuda_graphs.py::TestVisionTECudaGraphHelperPP2::test_pp2_create_cudagraphs_last_stage_noop` was observed hanging during CI on branch `ko3n1g/bump-te-release-v2.14-dev`.
The test sets up a PP=2 distributed process group (`Utils.initialize_model_parallel(pipeline_model_parallel_size=2)`). When running on a single-process runner the subsequent `torch.distributed.barrier()` inside `Utils.destroy_model_parallel()` (or the session-scoped `cleanup` fixture in `conftest.py`) waits indefinitely for the second rank that never appears.
## Workaround applied
Marked the test `@pytest.mark.flaky` / `@pytest.mark.flaky_in_dev` in #4332.
## Fix needed
Investigate and fix the root cause — either gate PP=2 tests to only run when `WORLD_SIZE >= 2`, or mock/stub the distributed barrier in teardown for single-process runs.
## Linked PR
#4332
Contributor guide
Assessment
This issue has not been assessed yet.