modelscope / modelscope/ms-swift

微调InternVL3.5的video的最大帧数的设置问题

Open
#6,191 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

documentation
Dominant language
Python
Stars
15.7k
Forks
1.7k
Avg merge
1d 16h
Merged PRs (30d)
136

Description

[rank3]: File "/home/lsy/miniconda3/envs/swift/lib/python3.10/site-packages/transformers/models/internvl/modeling_internvl.py", line 708, in pixel_shuffle
[rank3]: vision_features = vision_features.view(
[rank3]: RuntimeError: shape '[498, 27, 13, 2048]' is invalid for input of size 371755008

return func(*args, **kwargs)

File "/mnt/data2/SyL/ms-swift/swift/llm/template/base.py", line 497, in encode
encoded = self._encode_truncated(chosen)
File "/mnt/data2/SyL/ms-swift/swift/llm/template/base.py", line 1214, in _encode_truncated
raise MaxLengthError(f'Current length of row({length}) is larger'
swift.llm.template.base.MaxLengthError: Current length of row(256210) is larger than the max_length(81420).
显示token的长度有几十万,设置的最大帧数好像无法生效

这是我的脚本nproc_per_node=4

CUDA_VISIBLE_DEVICES=4,5,6,7
MAX_PIXELS=1003520
VIDEO_MAX_PIXELS=50176
MAX_NUM=8
NPROC_PER_NODE=$nproc_per_node
swift sft
--model xxx/InternVL3.5-4B-hf
--model_type internvl_hf
--train_type lora
--dataset /mnt/data2/SyL/ms-swift/datasets/xxx.json
--torch_dtype bfloat16
--num_train_epochs 1
--per_device_train_batch_size 2
--learning_rate 2e-4
--lora_rank 8
--lora_alpha 32
--target_modules all-linear
--gradient_accumulation_steps 4
--save_steps 500
--save_total_limit 4
--logging_steps 5
--max_length 81420
--output_dir out
--warmup_ratio 0.05
--dataloader_num_workers 4
--model_author swift
--model_name swift-robot
--deepspeed zero2

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the InternVL3.5 training command and inspect swift/llm/template/base.py, especially encode and _encode_truncated, alongside transformers/models/internvl/modeling_internvl.py at pixel_shuffle. Trace how VIDEO_MAX_PIXELS and MAX_NUM affect video tokens and compare that with the reported max_length failure; done means the configured frame or pixel limits take effect without the shape or length errors.

Written by the indexing model from the issue text.

Assessment

Tech stack
huggingface, python, pytorch
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.