NVIDIA / NVIDIA/Megatron-LM

LR decay being flat in the end

Open
#2,258 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
17.9k
Forks
4.5k
Avg merge
4d 6h
Merged PRs (30d)
271

Description

**Describe the bug**

LR will be flat in the end of decay, seems to be equal with the number of warming up steps.

**Steps/Code to reproduce bug**
When using, for example
TRAIN_SAMPLES=102400000
LR_WARMUP_SAMPLES=5120000
LR_DECAY_SAMPLES=$((TRAIN_SAMPLES-LR_WARMUP_SAMPLES))
--train-samples $TRAIN_SAMPLES \
--lr-decay-samples $LR_DECAY_SAMPLES \
--lr-warmup-samples $LR_WARMUP_SAMPLES \

**Additional context**

Image

Image

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.