[BUG] Full validation fails with a single dataset
- Dominant language
- Python
- Stars
- 17.9k
- Forks
- 4.5k
- Avg merge
- 4d 6h
- Merged PRs (30d)
- 271
Description
**Describe the bug**
On upstream `c6be9199750845bd99a7d1a438862bb9ab29b630`, full validation with one dataset raises `TypeError: 'int' object is not subscriptable` before evaluation. This does not require Engram.
**Steps/Code to reproduce bug**
Set `full_validation=True`, `multiple_validation_sets=False`, and `eval_iters=3` on the TP source rank.
The [upstream dispatch](https://github.com/NVIDIA/Megatron-LM/blob/c6be9199750845bd99a7d1a438862bb9ab29b630/megatron/training/training.py#L5490-L5507) normalizes the count to a list, but broadcasts a tensor created from the original scalar. This CPU snippet isolates the type mismatch; it does not reproduce distributed evaluation:
```python
import torch
count = 3
eval_iters = [count]
eval_iters = torch.tensor(count).tolist() # 3, not [3]
eval_iters[0] # TypeError
```
**Expected behavior**
Broadcast `[3]` and pass `3` to the evaluator. Constructing the tensor from normalized `eval_iters` preserves the required list shape after `.tolist()`.
**Additional context**
Local tests on 8×H800 used TP=2, real NCCL broadcast and the native dispatch with a mocked evaluator. The single-dataset case failed upstream; the `[2, 3]` two-dataset control passed. Both passed with the one-line fix.
Environment: Python 3.12.3, PyTorch `2.12.0a0+5aff3928d8.nv26.05`, CUDA 13.2. This checks dispatch, not complete dataset evaluation or official CI. The full regression tests will accompany the fix PR.
[#3388](https://github.com/NVIDIA/Megatron-LM/pull/3388) addresses a multiple-dataset deadlock; this report concerns scalar/list conversion for one dataset.
@NVIDIA/mcore-oncall
Contributor guide
Assessment
This issue has not been assessed yet.