NVIDIA / NVIDIA/Megatron-LM

tests: test_pp2_create_cudagraphs_last_stage_noop hangs intermittently

Open
#4,346 0 comments 0 reactions 1 assignee Claimed by @tomlifu View on GitHub
Dominant language
Python
Stars
17.9k
Forks
4.5k
Avg merge
4d 6h
Merged PRs (30d)
271

Description

## Description

`tests/unit_tests/transformer/test_vision_cuda_graphs.py::TestVisionTECudaGraphHelperPP2::test_pp2_create_cudagraphs_last_stage_noop` was observed hanging during CI on branch `ko3n1g/bump-te-release-v2.14-dev`.

The test sets up a PP=2 distributed process group (`Utils.initialize_model_parallel(pipeline_model_parallel_size=2)`). When running on a single-process runner the subsequent `torch.distributed.barrier()` inside `Utils.destroy_model_parallel()` (or the session-scoped `cleanup` fixture in `conftest.py`) waits indefinitely for the second rank that never appears.

## Workaround applied

Marked the test `@pytest.mark.flaky` / `@pytest.mark.flaky_in_dev` in #4332.

## Fix needed

Investigate and fix the root cause — either gate PP=2 tests to only run when `WORLD_SIZE >= 2`, or mock/stub the distributed barrier in teardown for single-process runs.

## Linked PR

#4332

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.