[BUG] GPTDataset forcibly replaces pad_token_id with 0, causinginference mismatch for models with pad_token_id != 0
- Dominant language
- Python
- Stars
- 17.9k
- Forks
- 4.5k
- Avg merge
- 4d 6h
- Merged PRs (30d)
- 271
Description
**Describe the bug**
In `megatron/core/datasets/gpt_dataset.py`, the `GPTDataset.__getitem__` method forcibly replaces `pad_token_id` with `0` in both `tokens` and `labels` tensors ([lines 214-215](https://github.com/NVIDIA/Megatron-LM/blob/23a1dcaf740ec5b9d667cb4968af327a2e1ad238/megatron/core/datasets/gpt_dataset.py#L206-L207)). This causes a training/inference mismatch for models where `pad_token_id` is not `0`, such as Qwen and DeepSeek models where `pad_token_id` equals `eos_token_id` (typically `1` or another non-zero value).
**Impact:**
- During training, the model learns that token ID `0` represents end-of-sequence because all `pad_token_id` values are replaced with `0`
- During inference, the tokenizer uses the actual `eos_token_id` (e.g., `1`), but the model was trained to recognize `0` as the end-of-sequence marker
- This causes the model to fail to properly recognize end-of-sequence tokens during inference, leading to incorrect generation behavior
Contributor guide
Assessment
This issue has not been assessed yet.