modelscope / modelscope/ms-swift
4*A800 LoRA 微调 Qwen3.5-4B（图片文本识别，seq_len=32k）：训练几步后单卡显存占满 OOM，同配置 Qwen3-VL 无此问题
Open
Nobody has claimed this yet.
question
- Dominant language
- Python
- Stars
- 15.7k
- Forks
- 1.7k
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 136
Description
Checklist / 检查清单
- I have searched existing issues, and this is a new question or discussion topic. / 我已经搜索过现有的 issues，确认这是一个新的问题与讨论。
Question Description / 问题描述
问题
GPU: 4×A800
模型: Qwen3.5-4B
微调方式: LoRA
任务: 文字识别
序列长度: 总长度限定 32k
核心问题: 采用官方推荐的微调脚本与配置参数，训练初期（几步内）出现单卡显存陡增并占满，最终触发 OOM 报错；相同数据集、相同软硬件配置下微调 Qwen3-VL 可稳定运行，无显存异常。
该现象是否由 Qwen3.5-4B 本身对 32k 长序列的显存需求是否显著更高？
还是环境的问题？
训练脚本
PYTORCH_CUDA_ALLOC_CONF='expandable_segments:True' \
IMAGE_MAX_TOKEN_NUM=16000 \
VIDEO_MAX_TOKEN_NUM=128 \
FPS_MAX_FRAMES=16 \
NPROC_PER_NODE=4 \
CUDA_VISIBLE_DEVICES=0,1,2,3 \
swift sft \
--model /Work/PTM/Qwen3.5/Qwen3.5-0.8B \
--dataset train_datasets.json \
--load_from_cache_file false \
--tuner_type lora \
--add_non_thinking_prefix true \
--loss_scale ignore_empty_think \
--torch_dtype bfloat16 \
--num_train_epochs 5 \
--per_device_train_batch_size 1 \
--per_device_eval_batch_size 1 \
--learning_rate 1e-4 \
--lora_rank 8 \
--lora_alpha 16 \
--target_modules all-linear \
--freeze_vit true \
--freeze_aligner true \
--vit_gradient_checkpointing false \
--gradient_accumulation_steps 1 \
--save_steps 500 \
--logging_steps 1 \
--max_length 32000 \
--output_dir qwen35-4B-lora \
--warmup_ratio 0.05 \
--deepspeed zero3 \
--dataset_num_proc 4 \
--dataloader_num_workers 4 \
--packing false \
--padding_free false
环境
Package Version Build
---------------------------------------- ------------- -----
absl-py 2.4.0
accelerate 1.13.0
addict 2.4.0
aiofiles 24.1.0
aiohappyeyeballs 2.6.1
aiohttp 3.13.4
aiosignal 1.4.0
aliyun-python-sdk-core 2.16.0
aliyun-python-sdk-kms 2.16.5
annotated-doc 0.0.4
annotated-types 0.7.0
anthropic 0.86.0
antlr4-python3-runtime 4.9.3
anyio 4.13.0
apache-tvm-ffi 0.1.9
astor 0.8.1
attrdict 2.0.1
attrs 26.1.0
av 17.0.0
binpacking 2.0.1
blake3 1.0.8
brotli 1.2.0
cachetools 7.0.5
causal_conv1d 1.6.1
cbor2 5.9.0
certifi 2026.2.25
cffi 2.0.0
charset-normalizer 3.4.6
click 8.3.1
cloudpickle 3.1.2
compressed-tensors 0.13.0
conda-pack 0.9.1
contourpy 1.3.3
cpm-kernels 1.0.11
crcmod 1.7
cryptography 46.0.6
cuda-bindings 12.9.4
cuda-pathfinder 1.5.0
cuda-python 12.9.4
cycler 0.12.1
dacite 1.9.2
datasets 3.6.0
deepspeed 0.18.8
depyf 0.20.0
dill 0.3.8
diskcache 5.6.3
distro 1.9.0
dnspython 2.8.0
docstring_parser 0.17.0
einops 0.8.2
email-validator 2.3.0
fastapi 0.135.2
fastapi-cli 0.0.24
fastapi-cloud-cli 0.15.1
fastar 0.9.0
ffmpy 1.0.0
filelock 3.25.2
fla-core 0.4.2
flash_attn 2.8.3
flash-linear-attention 0.4.2
flashinfer-python 0.6.6
fonttools 4.62.1
frozenlist 1.8.0
fsspec 2026.3.0
gguf 0.18.0
googleapis-common-protos 1.73.1
gradio 5.50.0
gradio_client 1.14.0
groovy 0.1.2
grpcio 1.78.0
h11 0.16.0
hf-xet 1.4.2
hjson 3.1.0
httpcore 1.0.9
httptools 0.7.1
httpx 0.28.1
httpx-sse 0.4.3
huggingface_hub 1.8.0
idna 3.11
ijson 3.5.0
importlib_metadata 8.7.1
interegular 0.3.3
Jinja2 3.1.6
jiter 0.13.0
jmespath 0.10.0
joblib 1.5.3
json_repair 0.58.7
jsonschema 4.26.0
jsonschema-specifications 2025.9.1
kiwisolver 1.5.0
lark 1.2.2
liger_kernel 0.7.0
llguidance 1.3.0
llvmlite 0.44.0
lm-format-enforcer 0.11.3
loguru 0.7.3
Markdown 3.10.2
markdown-it-py 4.0.0
MarkupSafe 3.0.3
matplotlib 3.10.8
mcp 1.26.0
mdurl 0.1.2
mistral_common 1.10.0
model-hosting-container-standards 0.1.14
modelscope 1.35.3
mpmath 1.3.0
ms_swift 4.0.3
msgpack 1.1.2
msgspec 0.20.0
multidict 6.7.1
multiprocess 0.70.16
networkx 3.6.1
ninja 1.13.0
nltk 3.9.4
numba 0.61.2
numpy 2.4.4
nvidia-cublas-cu12 12.8.4.1
nvidia-cuda-cupti-cu12 12.8.90
nvidia-cuda-nvrtc-cu12 12.8.93
nvidia-cuda-runtime-cu12 12.8.90
nvidia-cudnn-cu12 9.10.2.21
nvidia-cudnn-frontend 1.18.0
nvidia-cufft-cu12 11.3.3.83
nvidia-cufile-cu12 1.13.1.3
nvidia-curand-cu12 10.3.9.90
nvidia-cusolver-cu12 11.7.3.90
nvidia-cusparse-cu12 12.5.8.93
nvidia-cusparselt-cu12 0.7.1
nvidia-cutlass-dsl 4.4.2
nvidia-cutlass-dsl-libs-base 4.4.2
nvidia-ml-py 13.595.45
nvidia-nccl-cu12 2.27.5
nvidia-nvjitlink-cu12 12.8.93
nvidia-nvshmem-cu12 3.4.5
nvidia-nvtx-cu12 12.8.90
omegaconf 2.3.0
openai 2.24.0
openai-harmony 0.0.8
opencv-python-headless 4.13.0.92
opentelemetry-api 1.40.0
opentelemetry-exporter-otlp 1.40.0
opentelemetry-exporter-otlp-proto-common 1.40.0
opentelemetry-exporter-otlp-proto-grpc 1.40.0
opentelemetry-exporter-otlp-proto-http 1.40.0
opentelemetry-proto 1.40.0
opentelemetry-sdk 1.40.0
opentelemetry-semantic-conventions 0.61b0
opentelemetry-semantic-conventions-ai 0.5.1
orjson 3.11.7
oss2 2.19.1
outlines_core 0.2.11
packaging 26.0
pandas 2.3.3
partial-json-parser 0.2.1.1.post7
peft 0.18.1
pillow 11.3.0
pip 26.0.1
prometheus_client 0.24.1
prometheus-fastapi-instrumentator 7.1.0
propcache 0.4.1
protobuf 6.33.6
psutil 7.2.2
py-cpuinfo 9.0.0
pyarrow 23.0.1
pybase64 1.4.3
pycountry 26.2.16
pycparser 3.0
pycryptodome 3.23.0
pydantic 2.12.3
pydantic_core 2.41.4
pydantic-extra-types 2.11.1
pydantic-settings 2.13.1
pydub 0.25.1
Pygments 2.20.0
PyJWT 2.12.1
pyparsing 3.3.2
python-dateutil 2.9.0.post0
python-dotenv 1.2.2
python-json-logger 4.1.0
python-multipart 0.0.22
pytz 2026.1.post1
PyYAML 6.0.3
pyzmq 27.1.0
quack-kernels 0.3.7
qwen-vl-utils 0.0.14
referencing 0.37.0
regex 2026.3.32
requests 2.33.0
rich 14.3.3
rich-toolkit 0.19.7
rignore 0.7.6
rouge 1.0.1
rpds-py 0.30.0
ruff 0.15.8
safehttpx 0.1.7
safetensors 0.7.0
scipy 1.17.1
semantic-version 2.10.0
sentencepiece 0.2.1
sentry-sdk 2.56.0
setproctitle 1.3.7
setuptools 80.10.2
shellingham 1.5.4
simplejson 3.20.2
six 1.17.0
sniffio 1.3.1
sortedcontainers 2.4.0
sse-starlette 3.3.4
starlette 0.52.1
supervisor 4.3.0
sympy 1.14.0
tabulate 0.10.0
tensorboard 2.20.0
tensorboard-data-server 0.7.2
tiktoken 0.12.0
tokenizers 0.22.2
tomlkit 0.13.3
torch 2.10.0 3
torch_c_dlpack_ext 0.1.5
torchaudio 2.10.0
torchvision 0.25.0
tqdm 4.67.3
transformers 5.2.0
transformers-stream-generator 0.0.5
triton 3.6.0
trl 0.29.1
typer 0.24.1
typer-slim 0.24.0
typing_extensions 4.15.0
typing-inspection 0.4.2
tzdata 2025.3
urllib3 2.6.3
uv 0.11.2
uvicorn 0.42.0
uvloop 0.22.1
vllm 0.18.0
watchfiles 1.1.1
websockets 15.0.1
Werkzeug 3.1.7
wheel 0.46.3
xgrammar 0.1.33
xxhash 3.6.0
yarl 1.23.0
zipp 3.23.0
zstandard 0.25.0
显卡信息
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 570.133.20 Driver Version: 570.133.20 CUDA Version: 12.8 |
|-----------------------------------------+------------------------+----------------------+
报错信息：
[INFO:swift] Successfully registered post_encode hook: ['PeftModelForCausalLM'].
The tokenizer has new PAD/BOS/EOS tokens that differ from the model config and generation config. The model config and generation config were aligned accordingly, being updated with the tokenizer's values. Updated tokens: {'eos_token_id': 248046, 'pad_token_id': 248044}.
The tokenizer has new PAD/BOS/EOS tokens that differ from the model config and generation config. The model config and generation config were aligned accordingly, being updated with the tokenizer's values. Updated tokens: {'eos_token_id': 248046, 'pad_token_id': 248044}.
The tokenizer has new PAD/BOS/EOS tokens that differ from the model config and generation config. The model config and generation config were aligned accordingly, being updated with the tokenizer's values. Updated tokens: {'eos_token_id': 248046, 'pad_token_id': 248044}.
The tokenizer has new PAD/BOS/EOS tokens that differ from the model config and generation config. The model config and generation config were aligned accordingly, being updated with the tokenizer's values. Updated tokens: {'eos_token_id': 248046, 'pad_token_id': 248044}.
Train: 0%| | 0/5905 [00:00<?, ?it/s][INFO:swift] use_logits_to_keep: False
{'loss': '0.4106', 'grad_norm': '0.7492', 'learning_rate': '3.4e-07', 'token_acc': '0.8632', 'epoch': '0.0008467', 'global_step/max_steps': '1/5905', 'elapsed_time': '5s', 'remaining_time': '7h 23m 15s', 'memory(GiB)': '44.45', 'train_speed(s/it)': '4.505'}
{'loss': '0.4235', 'grad_norm': '0.6308', 'learning_rate': '6.8e-07', 'token_acc': '0.8562', 'epoch': '0.001693', 'global_step/max_steps': '2/5905', 'elapsed_time': '24s', 'remaining_time': '19h 25m 35s', 'memory(GiB)': '44.47', 'train_speed(s/it)': '11.85'}
{'loss': '0.4163', 'grad_norm': '0.6002', 'learning_rate': '1.01e-06', 'token_acc': '0.863', 'epoch': '0.00254', 'global_step/max_steps': '3/5905', 'elapsed_time': '25s', 'remaining_time': '13h 51m 41s', 'memory(GiB)': '47.62', 'train_speed(s/it)': '8.455'}
{'loss': '0.3727', 'grad_norm': '0.5798', 'learning_rate': '1.35e-06', 'token_acc': '0.8708', 'epoch': '0.003387', 'global_step/max_steps': '4/5905', 'elapsed_time': '28s', 'remaining_time': '11h 22m 46s', 'memory(GiB)': '78.12', 'train_speed(s/it)': '6.942'}
{'loss': '0.4318', 'grad_norm': '0.5082', 'learning_rate': '1.69e-06', 'token_acc': '0.8621', 'epoch': '0.004234', 'global_step/max_steps': '5/5905', 'elapsed_time': '31s', 'remaining_time': '10h 6m 46s', 'memory(GiB)': '78.12', 'train_speed(s/it)': '6.171'}
{'loss': '0.3917', 'grad_norm': '0.6419', 'learning_rate': '2.03e-06', 'token_acc': '0.868', 'epoch': '0.00508', 'global_step/max_steps': '6/5905', 'elapsed_time': '32s', 'remaining_time': '8h 50m 11s', 'memory(GiB)': '78.12', 'train_speed(s/it)': '5.393'}
{'loss': '0.4145', 'grad_norm': '0.6385', 'learning_rate': '2.36e-06', 'token_acc': '0.866', 'epoch': '0.005927', 'global_step/max_steps': '7/5905', 'elapsed_time': '34s', 'remaining_time': '8h 3m 56s', 'memory(GiB)': '78.12', 'train_speed(s/it)': '4.923'}
Train: 0%|▏ | 7/5905 [00:34<5:09:56, 3.15s/it][INFO:swift] last_model_checkpoint: None
[INFO:swift] best_model_checkpoint: None
[INFO:swift] images_dir: xx/qwen35-4B-lora/v17-20260409-133239/images
[rank0]: Traceback (most recent call last):
[rank0]: File "/Work/xx/anaconda3/envs/swift40/lib/python3.11/site-packages/swift/cli/sft.py", line 20, in <module>
[rank0]: sft_main()
[rank0]: File "/Work/xx/anaconda3/envs/swift40/lib/python3.11/site-packages/swift/pipelines/train/sft.py", line 354, in sft_main
[rank0]: return SwiftSft(args).main()
[rank0]: ^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/Work/xx/anaconda3/envs/swift40/lib/python3.11/site-packages/swift/pipelines/base.py", line 52, in main
[rank0]: result = self.run()
[rank0]: ^^^^^^^^^^
[rank0]: File "/Work/xx/anaconda3/envs/swift40/lib/python3.11/site-packages/swift/ray/base.py", line 168, in wrapper
[rank0]: return func(self, *args, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/Work/xx/anaconda3/envs/swift40/lib/python3.11/site-packages/swift/pipelines/train/sft.py", line 197, in run
[rank0]: return self.train(trainer)
[rank0]: ^^^^^^^^^^^^^^^^^^^
[rank0]: File "/Work/xx/anaconda3/envs/swift40/lib/python3.11/site-packages/swift/pipelines/train/sft.py", line 270, in train
[rank0]: trainer.train(resume_checkpoint)
[rank0]: File "/Work/xx/anaconda3/envs/swift40/lib/python3.11/site-packages/swift/trainers/mixin.py", line 895, in train
[rank0]: res = super().train(*args, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/Work/xx/anaconda3/envs/swift40/lib/python3.11/site-packages/transformers/trainer.py", line 1412, in train
[rank0]: return inner_training_loop(
[rank0]: ^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/Work/xx/anaconda3/envs/swift40/lib/python3.11/site-packages/transformers/trainer.py", line 1742, in _inner_training_loop
[rank0]: tr_loss_step = self.training_step(model, inputs, num_items_in_batch)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/Work/xx/anaconda3/envs/swift40/lib/python3.11/site-packages/swift/trainers/seq2seq_trainer.py", line 228, in training_step
[rank0]: return super().training_step(model, inputs, *args, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/Work/xx/anaconda3/envs/swift40/lib/python3.11/site-packages/transformers/trainer.py", line 1979, in training_step
[rank0]: self.accelerator.backward(loss, **kwargs)
[rank0]: File "/Work/xx/anaconda3/envs/swift40/lib/python3.11/site-packages/accelerate/accelerator.py", line 2830, in backward
[rank0]: self.deepspeed_engine_wrapped.backward(loss, sync_gradients=self.sync_gradients, **kwargs)
[rank0]: File "/Work/xx/anaconda3/envs/swift40/lib/python3.11/site-packages/accelerate/utils/deepspeed.py", line 270, in backward
[rank0]: self.engine.backward(loss, **kwargs)
[rank0]: File "/Work/xx/anaconda3/envs/swift40/lib/python3.11/site-packages/deepspeed/utils/nvtx.py", line 20, in wrapped_fn
[rank0]: ret_val = func(*args, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/Work/xx/anaconda3/envs/swift40/lib/python3.11/site-packages/deepspeed/runtime/engine.py", line 2583, in backward
[rank0]: loss.backward(**backward_kwargs)
[rank0]: File "/Work/xx/anaconda3/envs/swift40/lib/python3.11/site-packages/torch/_tensor.py", line 630, in backward
[rank0]: torch.autograd.backward(
[rank0]: File "/Work/xx/anaconda3/envs/swift40/lib/python3.11/site-packages/torch/autograd/__init__.py", line 364, in backward
[rank0]: _engine_run_backward(
[rank0]: File "/Work/xx/anaconda3/envs/swift40/lib/python3.11/site-packages/torch/autograd/graph.py", line 865, in _engine_run_backward
[rank0]: return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 26.50 GiB. GPU 0 has a total capacity of 79.25 GiB of which 19.75 GiB is free. Including non-PyTorch memory, this process has 59.48 GiB memory in use. Of the allocated memory 58.50 GiB is allocated by PyTorch, and 206.22 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
Train: 0%|▏ | 7/5905 [01:23<19:38:24, 11.99s/it]
W0409 13:34:14.568000 1561385 anaconda3/envs/swift40/lib/python3.11/site-packages/torch/distributed/elastic/multiprocessing/api.py:1010] Sending process 1561536 closing signal SIGTERM
W0409 13:34:14.568000 1561385 anaconda3/envs/swift40/lib/python3.11/site-packages/torch/distributed/elastic/multiprocessing/api.py:1010] Sending process 1561537 closing signal SIGTERM
W0409 13:34:14.569000 1561385 anaconda3/envs/swift40/lib/python3.11/site-packages/torch/distributed/elastic/multiprocessing/api.py:1010] Sending process 1561538 closing signal SIGTERM
E0409 13:34:15.800000 1561385 anaconda3/envs/swift40/lib/python3.11/site-packages/torch/distributed/elastic/multiprocessing/api.py:984] failed (exitcode: 1) local_rank: 0 (pid: 1561535) of binary: /Work/xx/anaconda3/envs/swift40/bin/python3.11
Traceback (most recent call last):
File "<frozen runpy>", line 198, in _run_module_as_main
File "<frozen runpy>", line 88, in _run_code
File "/Work/xx/anaconda3/envs/swift40/lib/python3.11/site-packages/torch/distributed/run.py", line 995, in <module>
main()
File "/Work/xx/anaconda3/envs/swift40/lib/python3.11/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 362, in wrapper
return f(*args, **kwargs)
^^^^^^^^^^^^^^^^^^
File "/Work/xx/anaconda3/envs/swift40/lib/python3.11/site-packages/torch/distributed/run.py", line 991, in main
run(args)
File "/Work/xx/anaconda3/envs/swift40/lib/python3.11/site-packages/torch/distributed/run.py", line 982, in run
elastic_launch(
File "/Work/xx/anaconda3/envs/swift40/lib/python3.11/site-packages/torch/distributed/launcher/api.py", line 170, in __call__
return launch_agent(self._config, self._entrypoint, list(args))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Work/xx/anaconda3/envs/swift40/lib/python3.11/site-packages/torch/distributed/launcher/api.py", line 317, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
============================================================
/Work/xx/anaconda3/envs/swift40/lib/python3.11/site-packages/swift/cli/sft.py FAILED
------------------------------------------------------------
Failures:
[1]:
time : 2026-04-09_13:34:15
host : hxrj
rank : 1 (local_rank: 1)
exitcode : -15 (pid: 1561536)
error_file: <N/A>
traceback : Signal 15 (SIGTERM) received by PID 1561536
[2]:
time : 2026-04-09_13:34:15
host : hxrj
rank : 2 (local_rank: 2)
exitcode : -15 (pid: 1561537)
error_file: <N/A>
traceback : Signal 15 (SIGTERM) received by PID 1561537
[3]:
time : 2026-04-09_13:34:15
host : hxrj
rank : 3 (local_rank: 3)
exitcode : -15 (pid: 1561538)
error_file: <N/A>
traceback : Signal 15 (SIGTERM) received by PID 1561538
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
time : 2026-04-09_13:34:14
host : hxrj
rank : 0 (local_rank: 0)
exitcode : 1 (pid: 1561535)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by running the provided swift sft command with the listed Qwen3.5-4B, LoRA, image-token, and max_length=32000 settings while monitoring per-card memory. Compare the behavior with the reported Qwen3-VL configuration; done means identifying whether the growth is caused by the model, sequence or image-token settings, or the environment, and documenting a reproducible explanation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 42/100