modelscope / modelscope/ms-swift
qwen2.5-vl与qwen3-vl同一份数据token长度差异巨大
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 15.7k
- Forks
- 1.7k
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 136
Description
在lora微调qwen2.5-vl-3b和qwen-3-vl-4b时使用同一个数据集,qwen3-vl的token会更长,具体如下
File "/usr/local/lib/python3.12/site-packages/swift/llm/dataset/utils.py", line 97, in __getitem__ return self.encode_func(data, return_length=True) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/swift/llm/template/base.py", line 497, in encode encoded = self._encode_truncated(chosen) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/swift/llm/template/base.py", line 1213, in _encode_truncated raise MaxLengthError(f'Current length of row({length}) is larger' swift.llm.template.base.MaxLengthError: Current length of row(36063) is larger than the max_length(20000).
下面是训练脚本:
` export nproc_per_node=2
export CUDA_VISIBLE_DEVICES=0,1
export MAX_PIXELS=1048576
export OMP_NUM_THREADS=8
export NPROC_PER_NODE=$nproc_per_node
swift sft
--model /mnt/data/vl-models/Qwen3-VL-4B-Instruct
--model_type qwen3_vl
--train_type lora
--dataset /mnt/workspace/vl-train/data/train6_with_negatives.jsonl
--val_dataset /mnt/workspace/vl-train/data/val6_with_negatives.jsonl
--temperature 0.7
--top_p 0.8
--top_k 20
--repetition_penalty=1.0
--torch_dtype bfloat16
--num_train_epochs 3
--per_device_train_batch_size 2
--per_device_eval_batch_size 2
--learning_rate 1e-4
--num_beams 1
--lora_rank 8
--lora_alpha 16
--target_modules all-linear
--freeze_vit true
--freeze_aligner true
--gradient_accumulation_steps 4
--eval_steps 5
--save_steps 5
--early_stop_interval 10
--save_total_limit 3
--logging_steps 1
--max_length 20000
--output_dir /mnt/workspace/vl-train/train_output
--warmup_ratio 0.05
--acc_strategy seq
--attn_impl flash_attn
--padding_free false
--dataloader_num_workers 2
--deepspeed zero2
--use_liger_kernel true `
两个模型只有model和model_type改了,其他不变。
qwen3-vl和qwen2.5-vl的token数怎么会相差这么大
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with swift/llm/dataset/utils.py::getitem and swift/llm/template/base.py::__encode_truncated, using the supplied swift sft command and dataset to reproduce the MaxLengthError. Compare the qwen2.5_vl and qwen3_vl encoding paths and model-specific template or tokenization behavior. Done means identifying the reason for the length difference and documenting or fixing the affected behavior with a reproducible check.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100