NVIDIA / NVIDIA/Megatron-LM

[BUG] Cannot use checkpoint to set learning rate

Open
#1,533 2 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
17.9k
Forks
4.5k
Avg merge
4d 6h
Merged PRs (30d)
271

Description

I continue to train my model from checkpoint. I use --use-checkpoint-opt_param-scheduler, but the learning rate starts from zero and not directly use learning saved in checkpoint.

some arguments:
--deterministic-mode \
--add-qkv-bias \
--use-distributed-optimizer ${BF16} \
--data-cache-path ${DATA_CACHE_PATH} \
--save-interval ${SAVE_INTERVAL} \
--seed ${SEED} ${NO_FP8_WGRAD} \
--use-mcore-models \
--data-path ${DATA_BLEND} \
--untie-embeddings-and-output-weights \
--disable-bias-linear \
--no-load-optim \
--no-load-rng \
--no-position-embedding \
--position-embedding-type rope \
--use-rotary-position-embeddings \
--rotary-percent 1 \
--rotary-base 1000000 \
--rotary-seq-len-interpolation-factor 1 \
--swiglu \
--attention-dropout 0.0 \
--hidden-dropout 0.0 \
--tensor-model-parallel-size ${TP_SIZE} \
--pipeline-model-parallel-size ${PP_SIZE} \
--context-parallel-size ${CP_SIZE} \
--sequence-parallel \
--num-layers ${NUM_LAYERS} \
--hidden-size ${HIDDEN_SIZE} \
--ffn-hidden-size ${FFN_HIDDEN_SIZE} \
--num-attention-heads ${ATT_HEADS} \
--seq-length 4096 \
--max-position-embeddings 4096 \
--normalization RMSNorm \
--norm-epsilon 1e-6 \
--micro-batch-size ${MBS} \
--global-batch-size ${GBS} \
--train-iters ${MAX_STEPS} \
--lr-decay-iters 8000000 \
--lr-warmup-iters 256 \
--lr ${LR} \
--min-lr 1e-5 \
--lr-decay-style cosine \
--log-interval 4 \
--eval-iters ${EVAL_ITERS} \
--eval-interval ${EVAL_INTERVAL} \
--tokenizer-type GPTSentencePieceTokenizer \
--tokenizer-model ${TOKENIZER} \
--split 99,1,0 \
--clip-grad 1.0 \
--weight-decay 0.1 \
--adam-beta1 0.9 \
--adam-beta2 0.95 \
--init-method-std 0.008 \
--log-throughput \
--log-timers-to-tensorboard \
--log-params-norm \
--log-num-zeros-in-grad \
--use-checkpoint-opt_param-scheduler \
--distributed-timeout-minutes 30 \
--save ${CHECKPOINT_DIR} \
--load ${CHECKPOINT_DIR}

Image

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.