Does `--resume_from` parameter matter?

Nobody has claimed this yet.

In our own training data, whether or not the --resume_from parameter is used will result in a significant difference in the initial loss. This is my training script with --resume_from parameter:

srun torchrun \
    --nnodes 2 \
    --nproc_per_node 8 \
    --node_rank $SLURM_NODEID \
    --rdzv_id $RANDOM \
    --rdzv_backend c10d \
    --rdzv_endpoint $head_node:29522 \
    pretrain_unified_navit.py \
    --num_shard 8 \
    --num_replicate 2 \
    --dataset_config_file ${DATA_CONFIG} \
    --model_path ${MODEL_PATH} \
    --layer_module Qwen2MoTDecoderLayer \
    --max_latent_size 64 \
    --total_steps 2000 \
    --save_every 1000 \
    --results_dir ${RESULTS_DIR} \
    --checkpoint_dir ${CHECKPOINTS} \
    --resume_from  ${MODEL_PATH}\
    --finetune_from_hf True \
    --auto_resume True \
    --resume-model-only True \
    --finetune-from-ema True \
    --log_every 1 \
    --lr 2e-5 \
    --num_worker 1 \
    --expected_num_tokens 10240 \
    --max_num_tokens 11520 \
    --max_num_tokens_per_sample 10240 \
    --vit_cond_dropout_prob 0 \
    --text_cond_dropout_prob 0

This is the training log with '--resume_from' parameter:

[[34m2025-06-03 22:47:42[0m] (step=0000002) Train Loss mse: 0.3486, Train Loss ce: 2.1082, Train Steps/Sec: 0.15, 
[[34m2025-06-03 22:47:48[0m] (step=0000003) Train Loss mse: 0.3899, Train Loss ce: 2.0838, Train Steps/Sec: 0.15, 
[[34m2025-06-03 22:47:55[0m] (step=0000004) Train Loss mse: 0.3445, Train Loss ce: 2.0912, Train Steps/Sec: 0.15, 
[[34m2025-06-03 22:48:01[0m] (step=0000005) Train Loss mse: 0.3639, Train Loss ce: 2.0755, Train Steps/Sec: 0.16, 
[[34m2025-06-03 22:48:08[0m] (step=0000006) Train Loss mse: 0.3715, Train Loss ce: 2.1182, Train Steps/Sec: 0.14,

This is the training log without '--resume_from' parameter:

[[34m2025-06-04 13:57:40[0m] (step=0000001) Train Loss mse: 1.9727, Train Loss ce: 12.7452, Train Steps/Sec: 0.16, 
[[34m2025-06-04 13:57:46[0m] (step=0000002) Train Loss mse: 1.9943, Train Loss ce: 12.7729, Train Steps/Sec: 0.15, 
[[34m2025-06-04 13:57:53[0m] (step=0000003) Train Loss mse: 1.8615, Train Loss ce: 12.7169, Train Steps/Sec: 0.15, 
[[34m2025-06-04 13:58:00[0m] (step=0000004) Train Loss mse: 1.9169, Train Loss ce: 12.7023, Train Steps/Sec: 0.15, 
[[34m2025-06-04 13:58:06[0m] (step=0000005) Train Loss mse: 1.8454, Train Loss ce: 12.6718, Train Steps/Sec: 0.15, 
...
[[34m2025-06-04 14:55:06[0m] (step=0000513) Train Loss mse: 1.3216, Train Loss ce: 1.8425, Train Steps/Sec: 0.15, 
[[34m2025-06-04 14:55:12[0m] (step=0000514) Train Loss mse: 1.3693, Train Loss ce: 1.9479, Train Steps/Sec: 0.15, 
[[34m2025-06-04 14:55:19[0m] (step=0000515) Train Loss mse: 1.3727, Train Loss ce: 2.0398, Train Steps/Sec: 0.15,

The model path and resume_from path are the same, and I set --resume-model-only True, why does this happen?

Start with pretrain_unified_navit.py and compare the argument-handling paths for --resume_from, --resume-model-only, --finetune_from_hf, and --finetune-from-ema. Reproduce the two runs using the supplied training commands and compare their initial logs. Done means explaining the loss difference and identifying whether the behavior requires a code or configuration change.

Written by the indexing model from the issue text.

Description

Contributor guide

First steps

Research direction

Assessment