qwen3-vl-4b sft保存checkpoint报错

问题
qwen3-vl-4b sft保存checkpoint报错

2026-01-09 17:40:26.041769 master-0 >> (min, max) time across ranks (ms):
2026-01-09 17:40:26.041804 master-0 >> evaluate .......................................: (130728.84, 130732.44)
2026-01-09 17:40:26.04396 master-0 >> WARNING:megatron.core.rerun_state_machine:Setting RerunStateMachine mode RerunMode.VALIDATE_RESULTS
2026-01-09 17:40:26.044628 master-0 >> -------------------------------------------------------------------------------------------------
2026-01-09 17:40:26.044634 master-0 >> validation loss at iteration 10000 | lm loss value: 3.893978E-01 | lm loss PPL: 1.476092E+00 |
2026-01-09 17:40:26.044636 master-0 >> -------------------------------------------------------------------------------------------------
2026-01-09 17:40:26.049709 master-0 >> saving checkpoint at iteration 10000 to /vlm-ssd/wanghanmo/e2e-问建筑/checkpoint/exp_all_0106a_q3_m_sft_3b_1600/v0-20260109-093839/checkpoint-10000 in torch_dist format
2026-01-09 17:40:26.050161 master-0 >> Storing distributed optimizer sharded state of type fully_sharded_model_space
2026-01-09 17:40:31.169213 master-0 >> [rank7]: Traceback (most recent call last):
2026-01-09 17:40:31.169245 master-0 >> [rank7]: File "/media/ms-swift/swift/cli/_megatron/sft.py", line 7, in
2026-01-09 17:40:31.169249 master-0 >> [rank7]: megatron_sft_main()
2026-01-09 17:40:31.16925 master-0 >> [rank7]: File "/media/ms-swift/swift/megatron/train/sft.py", line 87, in megatron_sft_main
2026-01-09 17:40:31.169252 master-0 >> [rank7]: return MegatronSft(args).main()
2026-01-09 17:40:31.169254 master-0 >> [rank7]: ^^^^^^^^^^^^^^^^^^^^^^^^
2026-01-09 17:40:31.169255 master-0 >> [rank7]: File "/media/ms-swift/swift/llm/base.py", line 49, in main
2026-01-09 17:40:31.169257 master-0 >> [rank7]: result = self.run()
2026-01-09 17:40:31.169258 master-0 >> [rank7]: ^^^^^^^^^^
2026-01-09 17:40:31.16926 master-0 >> [rank7]: File "/media/ms-swift/swift/megatron/train/sft.py", line 77, in run
2026-01-09 17:40:31.169261 master-0 >> [rank7]: self.trainer.train(train_dataset, val_dataset, data_collator)
2026-01-09 17:40:31.169263 master-0 >> [rank7]: File "/media/ms-swift/swift/megatron/trainers/base.py", line 1098, in train
2026-01-09 17:40:31.169264 master-0 >> [rank7]: pretrain(
2026-01-09 17:40:31.169266 master-0 >> [rank7]: File "/root/.cache/modelscope/_github/Megatron-LM/megatron/training/training.py", line 710, in pretrain
2026-01-09 17:40:31.169268 master-0 >> [rank7]: iteration, num_floating_point_operations_so_far = train(
2026-01-09 17:40:31.169269 master-0 >> [rank7]: ^^^^^^
2026-01-09 17:40:31.169271 master-0 >> [rank7]: File "/root/.cache/modelscope/_github/Megatron-LM/megatron/training/training.py", line 2261, in train
2026-01-09 17:40:31.169272 master-0 >> [rank7]: should_exit = checkpoint_and_decide_exit(
2026-01-09 17:40:31.169274 master-0 >> [rank7]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-01-09 17:40:31.169279 master-0 >> [rank7]: File "/root/.cache/modelscope/_github/Megatron-LM/megatron/training/training.py", line 1795, in checkpoint_and_decide_exit
2026-01-09 17:40:31.16928 master-0 >> [rank7]: save_checkpoint_and_time(
2026-01-09 17:40:31.169282 master-0 >> [rank7]: File "/root/.cache/modelscope/_github/Megatron-LM/megatron/training/training.py", line 1666, in save_checkpoint_and_time
2026-01-09 17:40:31.169283 master-0 >> [rank7]: save_checkpoint(
2026-01-09 17:40:31.169285 master-0 >> [rank7]: File "/media/ms-swift/swift/megatron/trainers/base.py", line 1038, in save_checkpoint
2026-01-09 17:40:31.169287 master-0 >> [rank7]: self._origin_save_checkpoint(iteration, model, *_args, **kwargs)
2026-01-09 17:40:31.169288 master-0 >> [rank7]: File "/root/.cache/modelscope/_github/Megatron-LM/megatron/training/checkpointing.py", line 531, in save_checkpoint
2026-01-09 17:40:31.16929 master-0 >> [rank7]: async_save_request = dist_checkpointing.save(state_dict, checkpoint_name, save_strategy,
2026-01-09 17:40:31.169292 master-0 >> [rank7]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-01-09 17:40:31.169294 master-0 >> [rank7]: File "/root/.cache/modelscope/_github/Megatron-LM/megatron/core/dist_checkpointing/serialization.py", line 425, in save
2026-01-09 17:40:31.169295 master-0 >> [rank7]: sharded_strategy.save(sharded_state_dict, checkpoint_dir)
2026-01-09 17:40:31.169297 master-0 >> [rank7]: File "/root/.cache/modelscope/_github/Megatron-LM/megatron/core/dist_checkpointing/strategies/fully_parallel.py", line 98, in save
2026-01-09 17:40:31.169298 master-0 >> [rank7]: return self.base_strategy.save(sharded_state_dict, checkpoint_dir)
2026-01-09 17:40:31.1693 master-0 >> [rank7]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-01-09 17:40:31.169301 master-0 >> [rank7]: File "/root/.cache/modelscope/_github/Megatron-LM/megatron/core/dist_checkpointing/strategies/base.py", line 223, in save
2026-01-09 17:40:31.169303 master-0 >> [rank7]: async_request = self.async_save(sharded_state_dict, checkpoint_dir)
2026-01-09 17:40:31.169305 master-0 >> [rank7]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-01-09 17:40:31.169306 master-0 >> [rank7]: File "/root/.cache/modelscope/_github/Megatron-LM/megatron/core/dist_checkpointing/strategies/torch.py", line 764, in async_save
2026-01-09 17:40:31.169307 master-0 >> [rank7]: ) = save_state_dict_async_plan(
2026-01-09 17:40:31.169309 master-0 >> [rank7]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-01-09 17:40:31.16931 master-0 >> [rank7]: File "/root/.cache/modelscope/_github/Megatron-LM/megatron/core/dist_checkpointing/strategies/state_dict_saver.py", line 141, in save_state_dict_async_plan
2026-01-09 17:40:31.169312 master-0 >> [rank7]: all_local_plans = dist_wrapper.gather_object(local_plan)
2026-01-09 17:40:31.169313 master-0 >> [rank7]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-01-09 17:40:31.169315 master-0 >> [rank7]: File "/usr/local/lib/python3.11/site-packages/torch/distributed/checkpoint/utils.py", line 135, in gather_object
2026-01-09 17:40:31.169318 master-0 >> [rank7]: dist.gather_object(
2026-01-09 17:40:31.169319 master-0 >> [rank7]: File "/usr/local/lib/python3.11/site-packages/torch/distributed/c10d_logger.py", line 81, in wrapper
2026-01-09 17:40:31.169321 master-0 >> [rank7]: return func(*args, **kwargs)
2026-01-09 17:40:31.169322 master-0 >> [rank7]: ^^^^^^^^^^^^^^^^^^^^^
2026-01-09 17:40:31.169324 master-0 >> [rank7]: File "/usr/local/lib/python3.11/site-packages/torch/distributed/distributed_c10d.py", line 3301, in gather_object
2026-01-09 17:40:31.169325 master-0 >> [rank7]: gather(
2026-01-09 17:40:31.169327 master-0 >> [rank7]: File "/usr/local/lib/python3.11/site-packages/torch/distributed/c10d_logger.py", line 81, in wrapper
2026-01-09 17:40:31.169328 master-0 >> [rank7]: return func(*args, **kwargs)
2026-01-09 17:40:31.16933 master-0 >> [rank7]: ^^^^^^^^^^^^^^^^^^^^^
2026-01-09 17:40:31.169331 master-0 >> [rank7]: File "/usr/local/lib/python3.11/site-packages/torch/distributed/distributed_c10d.py", line 4201, in gather
2026-01-09 17:40:31.169333 master-0 >> [rank7]: work = group.gather(output_tensors, input_tensors, opts)
2026-01-09 17:40:31.169334 master-0 >> [rank7]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-01-09 17:40:31.169336 master-0 >> [rank7]: RuntimeError: NCCL Error 1: unhandled cuda error (run with NCCL_DEBUG=INFO for details)
2026-01-09 17:41:12.871213 master-0 >> [rank7]:[W109 17:41:12.150540851 ProcessGroupNCCL.cpp:1538] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
2026-01-09 17:41:22.907089 master-0 >> W0109 17:41:22.906000 74 site-packages/torch/distributed/elastic/multiprocessing/api.py:900] Sending process 139 closing signal SIGTERM
2026-01-09 17:41:22.909905 master-0 >> W0109 17:41:22.909000 74 site-packages/torch/distributed/elastic/multiprocessing/api.py:900] Sending process 140 closing signal SIGTERM
2026-01-09 17:41:22.91552 master-0 >> W0109 17:41:22.915000 74 site-packages/torch/distributed/elastic/multiprocessing/api.py:900] Sending process 141 closing signal SIGTERM
2026-01-09 17:41:22.921237 master-0 >> W0109 17:41:22.920000 74 site-packages/torch/distributed/elastic/multiprocessing/api.py:900] Sending process 142 closing signal SIGTERM
2026-01-09 17:41:22.93505 master-0 >> W0109 17:41:22.924000 74 site-packages/torch/distributed/elastic/multiprocessing/api.py:900] Sending process 143 closing signal SIGTERM
2026-01-09 17:41:22.935066 master-0 >> W0109 17:41:22.929000 74 site-packages/torch/distributed/elastic/multiprocessing/api.py:900] Sending process 144 closing signal SIGTERM
2026-01-09 17:41:22.959408 master-0 >> W0109 17:41:22.959000 74 site-packages/torch/distributed/elastic/multiprocessing/api.py:900] Sending process 145 closing signal SIGTERM
2026-01-09 17:41:23.035634 master-0 >> E0109 17:41:23.035000 74 site-packages/torch/distributed/elastic/multiprocessing/api.py:874] failed (exitcode: 1) local_rank: 7 (pid: 146) of binary: /usr/local/bin/python
2026-01-09 17:41:23.037081 master-0 >> Traceback (most recent call last):
2026-01-09 17:41:23.037121 master-0 >> File "", line 198, in _run_module_as_main
2026-01-09 17:41:23.037124 master-0 >> File "", line 88, in _run_code
2026-01-09 17:41:23.037239 master-0 >> File "/usr/local/lib/python3.11/site-packages/torch/distributed/run.py", line 905, in
2026-01-09 17:41:23.037578 master-0 >> main()
2026-01-09 17:41:23.037581 master-0 >> File "/usr/local/lib/python3.11/site-packages/torch/distributed/elastic/multiprocessing/errors/init.py", line 357, in wrapper
2026-01-09 17:41:23.03771 master-0 >> return f(*args, **kwargs)
2026-01-09 17:41:23.038082 master-0 >> ^^^^^^^^^^^^^^^^^^
2026-01-09 17:41:23.038084 master-0 >> File "/usr/local/lib/python3.11/site-packages/torch/distributed/run.py", line 901, in main
2026-01-09 17:41:23.038349 master-0 >> run(args)
2026-01-09 17:41:23.038359 master-0 >> File "/usr/local/lib/python3.11/site-packages/torch/distributed/run.py", line 892, in run
2026-01-09 17:41:23.038733 master-0 >> elastic_launch(
2026-01-09 17:41:23.038736 master-0 >> File "/usr/local/lib/python3.11/site-packages/torch/distributed/launcher/api.py", line 143, in call
2026-01-09 17:41:23.038738 master-0 >> return launch_agent(self._config, self._entrypoint, list(args))
2026-01-09 17:41:23.039489 master-0 >> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-01-09 17:41:23.039509 master-0 >> File "/usr/local/lib/python3.11/site-packages/torch/distributed/launcher/api.py", line 277, in launch_agent
2026-01-09 17:41:23.039597 master-0 >> raise ChildFailedError(
2026-01-09 17:41:23.039684 master-0 >> torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
2026-01-09 17:41:23.039687 master-0 >> ============================================================
2026-01-09 17:41:23.039688 master-0 >> /media/ms-swift/swift/cli/_megatron/sft.py FAILED
2026-01-09 17:41:23.03969 master-0 >> ------------------------------------------------------------
2026-01-09 17:41:23.039692 master-0 >> Failures:
2026-01-09 17:41:23.039693 master-0 >> <NO_OTHER_FAILURES>

环境

absl-py 2.3.1
accelerate 1.12.0
addict 2.4.0
aiofiles 24.1.0
aiohappyeyeballs 2.6.1
aiohttp 3.13.2
aiosignal 1.4.0
airportsdata 20250909
aliyun-python-sdk-core 2.16.0
aliyun-python-sdk-kms 2.16.5
annotated-doc 0.0.4
annotated-types 0.7.0
anthropic 0.75.0
antlr4-python3-runtime 4.9.3
anyio 4.12.0
apache-tvm-ffi 0.1.4
apex 0.1
astor 0.8.1
asttokens 3.0.1
attrdict 2.0.1
attrs 25.4.0
audioread 3.1.0
auto_gptq 0.7.1
av 16.0.1
binpacking 1.5.2
bitsandbytes 0.48.2
blake3 1.0.8
blobfile 3.0.0
boto3 1.41.5
botocore 1.41.5
brotli 1.2.0
build 1.3.0
cachetools 6.2.2
cbor2 5.7.1
certifi 2025.11.12
cffi 2.0.0
cfgv 3.5.0
charset-normalizer 3.4.4
click 8.2.1
cloudpickle 3.1.2
colorama 0.4.6
colorlog 6.10.1
compressed-tensors 0.11.0
contourpy 1.3.3
cpm-kernels 1.0.11
crcmod 1.7
cryptography 46.0.3
cuda-bindings 13.0.3
cuda-pathfinder 1.3.2
cuda-python 13.0.3
cupy-cuda12x 13.6.0
cycler 0.12.1
dacite 1.9.2
datasets 3.6.0
decorator 5.2.1
decord 0.6.0
decord2 2.0.0
deepspeed 0.17.6
depyf 0.19.0
diffusers 0.35.2
dill 0.3.8
diskcache 5.6.3
distlib 0.4.0
distro 1.9.0
dnspython 2.8.0
docstring_parser 0.17.0
dotenv 0.9.9
editdistance 0.8.1
einops 0.8.1
email-validator 2.3.0
et_xmlfile 2.0.0
evalscope 1.3.0
evaluate 0.4.6
executing 2.2.1
fastapi 0.122.0
fastapi-cli 0.0.16
fastapi-cloud-cli 0.5.2
fastar 0.8.0
fastrlock 0.8.3
ffmpy 1.0.0
filelock 3.20.0
fire 0.7.1
flash-attn 2.7.4.post1
flashinfer-cubin 0.5.2
flashinfer-python 0.5.2
fonttools 4.61.0
frozendict 2.4.7
frozenlist 1.8.0
fsspec 2025.3.0
funasr 1.2.7
func-timeout 4.3.5
future 1.0.0
fuzzywuzzy 0.18.0
gekko 1.3.0
gguf 0.17.1
gitdb 4.0.12
GitPython 3.1.45
google-auth 2.43.0
google-genai 1.52.0
gradio 5.32.1
gradio_client 1.10.2
groovy 0.1.2
grpcio 1.75.1
grpcio-health-checking 1.75.1
grpcio-reflection 1.75.1
grpcio-tools 1.75.1
h11 0.16.0
h5py 3.15.1
hf_transfer 0.1.9
hf-xet 1.2.0
hjson 3.1.0
httpcore 1.0.9
httptools 0.7.1
httpx 0.28.1
huggingface-hub 0.36.0
human-eval 1.0.3
hydra-core 1.3.2
icecream 2.1.8
identify 2.6.15
idna 3.11
ImageIO 2.37.2
immutabledict 4.2.2
importlib_metadata 8.7.0
interegular 0.3.3
ipdb 0.13.13
ipython 9.7.0
ipython_pygments_lexers 1.1.1
jaconv 0.4.1
jamo 0.4.1
jedi 0.19.2
jieba 0.42.1
Jinja2 3.1.6
jiter 0.12.0
jmespath 0.10.0
joblib 1.5.2
json_repair 0.54.2
json5 0.12.1
jsonlines 4.0.0
jsonschema 4.25.1
jsonschema-specifications 2025.9.1
kaldiio 2.18.1
kiwisolver 1.4.9
langdetect 1.0.9
lark 1.2.2
latex2sympy2_extended 1.10.2
lazy_loader 0.4
Levenshtein 0.27.3
librosa 0.11.0
liger_kernel 0.6.4
lightning-utilities 0.15.2
llguidance 0.7.30
llvmlite 0.44.0
lm-format-enforcer 0.11.3
lmdeploy 0.10.2
lxml 6.0.2
Markdown 3.10
markdown-it-py 4.0.0
MarkupSafe 3.0.3
math-verify 0.8.0
matplotlib 3.10.7
matplotlib-inline 0.2.1
mdurl 0.1.2
megatron-core 0.14.1
mistral_common 1.8.5
ml_dtypes 0.5.4
mmengine-lite 0.10.7
modelscope 1.31.0
more-itertools 10.8.0
mpmath 1.3.0
ms-agent 1.5.1
ms-opencompass 0.1.6
ms-swift 3.13.0.dev0 /media/ms-swift
ms-vlmeval 0.0.19
msgpack 1.1.2
msgspec 0.20.0
multidict 6.7.0
multiprocess 0.70.16
nest-asyncio 1.6.0
networkx 3.6
ninja 1.13.0
nltk 3.9.2
nodeenv 1.9.1
numba 0.61.2
numpy 1.26.4
nvidia-cublas-cu12 12.8.4.1
nvidia-cuda-cupti-cu12 12.8.90
nvidia-cuda-nvrtc-cu12 12.8.93
nvidia-cuda-runtime-cu12 12.8.90
nvidia-cudnn-cu12 9.10.2.21
nvidia-cudnn-frontend 1.16.0
nvidia-cufft-cu12 11.3.3.83
nvidia-cufile-cu12 1.13.1.3
nvidia-curand-cu12 10.3.9.90
nvidia-cusolver-cu12 11.7.3.90
nvidia-cusparse-cu12 12.5.8.93
nvidia-cusparselt-cu12 0.7.1
nvidia-cutlass-dsl 4.2.1
nvidia-ml-py 13.580.82
nvidia-nccl-cu12 2.27.3
nvidia-nvjitlink-cu12 12.8.93
nvidia-nvtx-cu12 12.8.90
nvitop 1.6.0
omegaconf 2.3.0
onnx 1.19.1
onnx-ir 0.1.12
onnxscript 0.5.6
openai 2.6.1
openai-harmony 0.0.4
openai-whisper 20250625
OpenCC 1.1.9
opencv-python 4.11.0.86
opencv-python-headless 4.12.0.88
openpyxl 3.1.5
optimum 2.0.0
orjson 3.11.4
oss2 2.19.1
outlines 0.1.11
outlines_core 0.1.26
overrides 7.7.0
packaging 25.0
pandas 2.3.3
parso 0.8.5
partial-json-parser 0.2.1.1.post7
peft 0.17.1
pexpect 4.9.0
pillow 11.3.0
pip 23.3.2
platformdirs 4.5.0
pooch 1.8.2
portalocker 3.2.0
pre_commit 4.5.0
prettytable 3.17.0
prometheus_client 0.23.1
prometheus-fastapi-instrumentator 7.1.0
prompt_toolkit 3.0.52
propcache 0.4.1
protobuf 6.33.1
psutil 7.1.3
ptyprocess 0.7.0
pure_eval 0.2.3
py-cpuinfo 9.0.0
py-spy 0.4.1
pyarrow 22.0.0
pyasn1 0.6.1
pyasn1_modules 0.4.2
pybase64 1.4.2
pybind11 3.0.1
pycountry 24.6.1
pycparser 2.23
pycryptodome 3.23.0
pycryptodomex 3.23.0
pydantic 2.11.10
pydantic_core 2.33.2
pydantic-extra-types 2.10.6
pydub 0.25.1
pyecharts 2.0.9
Pygments 2.19.2
pylatexenc 2.10
pynndescent 0.5.13
pynvml 13.0.1
pyparsing 3.2.5
pypinyin 0.55.0
pyproject_hooks 1.2.0
python-dateutil 2.9.0.post0
python-dotenv 1.2.1
python-json-logger 4.0.0
python-Levenshtein 0.27.3
python-multipart 0.0.20
pytorch-wpe 0.0.1
pytz 2025.2
PyYAML 6.0.3
pyzmq 27.1.0
qwen-omni-utils 0.0.8
qwen-vl-utils 0.0.14
rank-bm25 0.2.2
RapidFuzz 3.14.3
ray 2.52.1
referencing 0.37.0
regex 2025.11.3
requests 2.32.5
rich 13.9.4
rich-toolkit 0.17.0
rignore 0.7.6
rouge 1.0.1
rouge-chinese 1.0.3
rouge-score 0.1.2
rpds-py 0.29.0
rsa 4.9.1
ruff 0.14.7
s3transfer 0.15.0
sacrebleu 2.5.1
safehttpx 0.1.7
safetensors 0.7.0
scikit-learn 1.7.2
scipy 1.16.3
seaborn 0.13.2
semantic-version 2.10.0
sentence-transformers 5.1.2
sentencepiece 0.2.1
sentry-sdk 2.46.0
setproctitle 1.3.7
setuptools 65.5.1
sgl-kernel 0.3.17.post1
sglang 0.5.5.post3
shellingham 1.5.4
shortuuid 1.0.13
simplejson 3.20.2
six 1.17.0
smmap 5.0.2
sniffio 1.3.1
sortedcontainers 2.4.0
soundfile 0.13.1
soxr 1.0.0
stack-data 0.6.3
starlette 0.50.0
sty 1.0.6
swanlab 0.7.2
sympy 1.14.0
tabulate 0.9.0
tenacity 9.1.2
tensorboard 2.20.0
tensorboard-data-server 0.7.2
tensorboardX 2.6.4
termcolor 3.2.0
threadpoolctl 3.6.0
tiktoken 0.12.0
timeout-decorator 0.5.0
timm 1.0.16
tokenizers 0.22.1
tomlkit 0.13.3
torch 2.8.0
torch-complex 0.4.4
torch_memory_saver 0.0.9
torchao 0.9.0
torchaudio 2.8.0
torchmetrics 1.8.2
torchvision 0.23.0
tqdm 4.67.1
traitlets 5.14.3
transformer_engine 2.9.0
transformer_engine_cu12 2.9.0
transformer-engine-torch 2.9.0
transformers 4.57.3
transformers-stream-generator 0.0.5
triton 3.4.0
trl 0.23.1
typer 0.20.0
typing_extensions 4.15.0
typing-inspection 0.4.2
tzdata 2025.2
umap-learn 0.5.9.post2
urllib3 2.5.0
uvicorn 0.38.0
uvloop 0.21.0
validators 0.35.0
virtualenv 20.35.4
vllm 0.11.0
wandb 0.23.0
watchfiles 1.1.1
wcwidth 0.2.14
websockets 15.0.1
Werkzeug 3.1.4
wheel 0.45.1
word2number 1.1
wrapt 2.0.1
xformers 0.0.32.post1
xgrammar 0.1.25
xlsxwriter 3.2.9
xxhash 3.6.0
yapf 0.43.0
yarl 1.22.0
zipp 3.23.0
zstandard 0.25.0

脚本

PYTORCH_CUDA_ALLOC_CONF='expandable_segments:True'
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7,8
IMAGE_MAX_TOKEN_NUM=1600
NPROC_PER_NODE=8
megatron sft
--model 'Qwen3-VL-4B-Instruct'
--model_type qwen3_vl
--load_safetensors true
--save_safetensors true
--context_parallel_size 1
--tensor_model_parallel_size 1
--pipeline_model_parallel_size 1
--dataset $SFT_DATASET
--split_dataset_ratio 0.01
--load_from_cache_file true
--bf16 true
--attention_softmax_in_fp32 true
--attention_backend flash
--sequence_parallel false
--packing false
--freeze_llm false
--freeze_vit false
--freeze_aligner false
--micro_batch_size 16
--global_batch_size 128
--recompute_granularity full
--recompute_method uniform
--recompute_num_layers 1
--finetune true
--cross_entropy_loss_fusion true
--train_type full
--lr 1e-5
--lr_warmup_fraction 0.05
--min_lr 1e-8
--max_epochs 1
--clip_grad 1.0
--save $SFT_DIR
--save_interval 10000
--eval_interval 10000
--vit_gradient_checkpointing true
--max_length 4096
--num_workers 4
--no_save_optim true
--no_save_rng true
--dataset_num_proc 8

Start with swift/megatron/trainers/base.py at save_checkpoint, then trace the reported call through Megatron's training/checkpointing.py and dist_checkpointing serialization.py. Reproduce the qwen3-vl-4b SFT checkpoint save and inspect the NCCL failure at torch.distributed.gather_object; done means the checkpoint saves successfully without the reported error.