modelscope / modelscope/ms-swift

qwen2.5-vl与qwen3-vl同一份数据token长度差异巨大

Open
#6,179 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement stale
Dominant language
Python
Stars
15.7k
Forks
1.7k
Avg merge
1d 16h
Merged PRs (30d)
136

Description

在lora微调qwen2.5-vl-3b和qwen-3-vl-4b时使用同一个数据集,qwen3-vl的token会更长,具体如下
File "/usr/local/lib/python3.12/site-packages/swift/llm/dataset/utils.py", line 97, in __getitem__ return self.encode_func(data, return_length=True) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/swift/llm/template/base.py", line 497, in encode encoded = self._encode_truncated(chosen) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/swift/llm/template/base.py", line 1213, in _encode_truncated raise MaxLengthError(f'Current length of row({length}) is larger' swift.llm.template.base.MaxLengthError: Current length of row(36063) is larger than the max_length(20000).

下面是训练脚本:
` export nproc_per_node=2
export CUDA_VISIBLE_DEVICES=0,1
export MAX_PIXELS=1048576
export OMP_NUM_THREADS=8
export NPROC_PER_NODE=$nproc_per_node

swift sft
--model /mnt/data/vl-models/Qwen3-VL-4B-Instruct
--model_type qwen3_vl
--train_type lora
--dataset /mnt/workspace/vl-train/data/train6_with_negatives.jsonl
--val_dataset /mnt/workspace/vl-train/data/val6_with_negatives.jsonl
--temperature 0.7
--top_p 0.8
--top_k 20
--repetition_penalty=1.0
--torch_dtype bfloat16
--num_train_epochs 3
--per_device_train_batch_size 2
--per_device_eval_batch_size 2
--learning_rate 1e-4
--num_beams 1
--lora_rank 8
--lora_alpha 16
--target_modules all-linear
--freeze_vit true
--freeze_aligner true
--gradient_accumulation_steps 4
--eval_steps 5
--save_steps 5
--early_stop_interval 10
--save_total_limit 3
--logging_steps 1
--max_length 20000
--output_dir /mnt/workspace/vl-train/train_output
--warmup_ratio 0.05
--acc_strategy seq
--attn_impl flash_attn
--padding_free false
--dataloader_num_workers 2
--deepspeed zero2
--use_liger_kernel true `

两个模型只有model和model_type改了,其他不变。

qwen3-vl和qwen2.5-vl的token数怎么会相差这么大

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with swift/llm/dataset/utils.py::getitem and swift/llm/template/base.py::__encode_truncated, using the supplied swift sft command and dataset to reproduce the MaxLengthError. Compare the qwen2.5_vl and qwen3_vl encoding paths and model-specific template or tokenization behavior. Done means identifying the reason for the length difference and documenting or fixing the affected behavior with a reproducible check.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.