When batch size is bigger than dataset size, the SFT finishes without error, but not doing any grad updates
- Dominant language
- Python
- Stars
- 2k
- Forks
- 561
- Avg merge
- 4d 5h
- Merged PRs (30d)
- 145
Description
**Describe the bug**
E.g. if bs=1024, but dataset only has 128 samples, I just see a bunch of these in the logs
========================= Epoch 1/2000 =========================
========================= Epoch 2/2000 =========================
========================= Epoch 3/2000 =========================
========================= Epoch 4/2000 =========================
========================= Epoch 5/2000 =========================
========================= Epoch 6/2000 =========================
========================= Epoch 7/2000 =========================
========================= Epoch 8/2000 =========================
and then the final checkpoint is saved but is the same as original.
**Steps/Code to reproduce bug**
Run any SFT with batch size bigger than dataset size
**Expected behavior**
Either fail with error or reduce batch size automatically to be equal to the dataset size.
Contributor guide
Assessment
This issue has not been assessed yet.