NVIDIA-NeMo / NVIDIA-NeMo/RL

When batch size is bigger than dataset size, the SFT finishes without error, but not doing any grad updates

Open
#921 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
2k
Forks
561
Avg merge
4d 5h
Merged PRs (30d)
145

Description

**Describe the bug**

E.g. if bs=1024, but dataset only has 128 samples, I just see a bunch of these in the logs

========================= Epoch 1/2000 =========================

========================= Epoch 2/2000 =========================

========================= Epoch 3/2000 =========================

========================= Epoch 4/2000 =========================

========================= Epoch 5/2000 =========================

========================= Epoch 6/2000 =========================

========================= Epoch 7/2000 =========================

========================= Epoch 8/2000 =========================

and then the final checkpoint is saved but is the same as original.

**Steps/Code to reproduce bug**

Run any SFT with batch size bigger than dataset size

**Expected behavior**

Either fail with error or reduce batch size automatically to be equal to the dataset size.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.