NVIDIA-NeMo / NVIDIA-NeMo/RL

main: L0_Unit_Tests_Megatron_Policy_1 fails, blocking functional tests on all PRs

Open
#4,035 0 comments 0 reactions 1 assignee Claimed by @terrykong View on GitHub
bug
Dominant language
Python
Stars
2k
Forks
561
Avg merge
4d 5h
Merged PRs (30d)
145

Description

This unit test has been failing on `main`:

```
FAILED test_megatron_split_state.py::TestTrainMicrobatch::
test_forwards_model_owned_packing_flags[model-owned-cp-slicing]
```

A guard in `train_microbatch` contradicts the test's parameters.

`megatron_policy_worker.py:1465`:

```python
if self.media_placeholder_token_id is not None or self.model_slices_context_parallel_inputs:
raise NotImplementedError("train_microbatch does not support multimodal models: ...")
```

`test_megatron_split_state.py:446`:

```python
pytest.param(False, False, True, id="model-owned-cp-slicing")
# ^^^^ model_slices_context_parallel_inputs=True
```

The test sets `model_slices_context_parallel_inputs` to `True` and asserts it is forwarded, but the guard raises as soon as it sees that flag set. The case cannot pass.

The guard was added by #2957 (`da39e1785`); the parameter case predates it.

Once the unit tests fail, CI skips the functional stage, so no PR based on `main` gets functional coverage. Our PR #3566 is blocked on this.

Could a small fix land, or the case be marked `xfail` for now?

@ZhiyuLi-Nvidia

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.