modelscope / modelscope/ms-swift
微调InternVL3.5的video的最大帧数的设置问题
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 15.7k
- Forks
- 1.7k
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 136
Description
[rank3]: File "/home/lsy/miniconda3/envs/swift/lib/python3.10/site-packages/transformers/models/internvl/modeling_internvl.py", line 708, in pixel_shuffle
[rank3]: vision_features = vision_features.view(
[rank3]: RuntimeError: shape '[498, 27, 13, 2048]' is invalid for input of size 371755008
return func(*args, **kwargs)
File "/mnt/data2/SyL/ms-swift/swift/llm/template/base.py", line 497, in encode
encoded = self._encode_truncated(chosen)
File "/mnt/data2/SyL/ms-swift/swift/llm/template/base.py", line 1214, in _encode_truncated
raise MaxLengthError(f'Current length of row({length}) is larger'
swift.llm.template.base.MaxLengthError: Current length of row(256210) is larger than the max_length(81420).
显示token的长度有几十万,设置的最大帧数好像无法生效
这是我的脚本nproc_per_node=4
CUDA_VISIBLE_DEVICES=4,5,6,7
MAX_PIXELS=1003520
VIDEO_MAX_PIXELS=50176
MAX_NUM=8
NPROC_PER_NODE=$nproc_per_node
swift sft
--model xxx/InternVL3.5-4B-hf
--model_type internvl_hf
--train_type lora
--dataset /mnt/data2/SyL/ms-swift/datasets/xxx.json
--torch_dtype bfloat16
--num_train_epochs 1
--per_device_train_batch_size 2
--learning_rate 2e-4
--lora_rank 8
--lora_alpha 32
--target_modules all-linear
--gradient_accumulation_steps 4
--save_steps 500
--save_total_limit 4
--logging_steps 5
--max_length 81420
--output_dir out
--warmup_ratio 0.05
--dataloader_num_workers 4
--model_author swift
--model_name swift-robot
--deepspeed zero2
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the InternVL3.5 training command and inspect swift/llm/template/base.py, especially encode and _encode_truncated, alongside transformers/models/internvl/modeling_internvl.py at pixel_shuffle. Trace how VIDEO_MAX_PIXELS and MAX_NUM affect video tokens and compare that with the reported max_length failure; done means the configured frame or pixel limits take effect without the shape or length errors.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- huggingface, python, pytorch
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100