megatron训练卡死

任务命令

export MODELSCOPE_CACHE='/model/shared'
export MEGATRON_LM_PATH='./swift/Megatron-LM'
PYTORCH_CUDA_ALLOC_CONF='expandable_segments:True'
NPROC_PER_NODE=8
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
IMAGE_MAX_TOKEN_NUM=1280
VIDEO_MAX_TOKEN_NUM=128
FPS_MAX_FRAMES=16
megatron sft
--load ./Qwen3-VL-30B-A3B-Instruct-mcore
--dataset test_5w.jsonl
--train_type full
--load_from_cache_file true
--split_dataset_ratio 0.01
--moe_permute_fusion true
--tensor_model_parallel_size 2
--expert_model_parallel_size 8
--sequence_parallel true
--freeze_vit false
--freeze_aligner false
--moe_grouped_gemm true
--moe_shared_expert_overlap true
--moe_aux_loss_coeff 1e-6
--micro_batch_size 4
--global_batch_size 96
--recompute_granularity full
--recompute_method uniform
--recompute_num_layers 1
--max_epochs 6
--finetune true
--cross_entropy_loss_fusion true
--lr 1e-5
--lr_warmup_fraction 0.05
--min_lr 1e-6
--save ./megatron_output/Qwen3-VL-30B-A3B-Instruct
--eval_interval 800
--save_interval 1621
--max_length 4096
--packing true
--num_workers 48
--dataset_num_proc 48
--no_save_optim true
--no_save_rng true
--moe_expert_capacity_factor 1
--optimizer_cpu_offload true
--use_precision_aware_optimizer true
--optimizer_offload_fraction 0.2
--attention_backend flash

训练卡死
WARNING:DotProductAttention:flash-attn v3 may provide important feature support or performance improvement. Please install flash-attn v3 by
(1) git clone https://github.com/Dao-AILab/flash-attention.git
(2) cd flash-attention/ && git checkout 3ba6f82 && git submodule update --init && cd hopper/ && python setup.py install
(3) python_path=python -c "import site; print(site.getsitepackages()[0])"
(4) mkdir -p $python_path/flash_attn_3
(5) cp flash_attn_interface.py $python_path/flash_attn_3/flash_attn_interface.py
Number of parameters in transformer block in billions: 29.90
Number of parameters in embedding layers in billions: 0.62
Total number of parameters in billions: 30.52
Number of parameters in most loaded shard in billions: 15.2599
Theoretical memory footprints: weight and optimizer=109147.05 MB
[Rank 0] (after 1 iterations) memory (MB) | allocated: 50183.49755859375 | max allocated: 54219.77099609375 | reserved: 56706.0 | max reserved: 56706.0
[Rank 1] (after 1 iterations) memory (MB) | allocated: 50183.49755859375 | max allocated: 54219.77099609375 | reserved: 56686.0 | max reserved: 56686.0

版本

使用两台机器，训练卡死，单机训练没有问题

Start by reproducing the posted megatron sft command across two machines, then inspect the logs around the first iteration and compare them with the single-machine run. Trace the distributed launch and synchronization point responsible for the hang; done means the multi-machine training either proceeds or the failure is documented with a reproducible diagnosis.