NVIDIA / NVIDIA/Megatron-LM

[BUG] GPTDataset forcibly replaces pad_token_id with 0, causinginference mismatch for models with pad_token_id != 0

Open
#2,170 1 comment 0 reactions 0 assignees View on GitHub
bug community-request
Dominant language
Python
Stars
17.9k
Forks
4.5k
Avg merge
4d 6h
Merged PRs (30d)
271

Description

**Describe the bug**

In `megatron/core/datasets/gpt_dataset.py`, the `GPTDataset.__getitem__` method forcibly replaces `pad_token_id` with `0` in both `tokens` and `labels` tensors ([lines 214-215](https://github.com/NVIDIA/Megatron-LM/blob/23a1dcaf740ec5b9d667cb4968af327a2e1ad238/megatron/core/datasets/gpt_dataset.py#L206-L207)). This causes a training/inference mismatch for models where `pad_token_id` is not `0`, such as Qwen and DeepSeek models where `pad_token_id` equals `eos_token_id` (typically `1` or another non-zero value).

**Impact:**
- During training, the model learns that token ID `0` represents end-of-sequence because all `pad_token_id` values are replaced with `0`
- During inference, the tokenizer uses the actual `eos_token_id` (e.g., `1`), but the model was trained to recognize `0` as the end-of-sequence marker
- This causes the model to fail to properly recognize end-of-sequence tokens during inference, leading to incorrect generation behavior

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.