modelscope / modelscope/ms-swift

Qwen3-VL SFT n_try_fetch error

Open
#7,845 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

question
Dominant language
Python
Stars
15.7k
Forks
1.7k
Avg merge
1d 16h
Merged PRs (30d)
136

Description

Checklist / 检查清单
  • I have searched existing issues, and this is a new question or discussion topic. / 我已经搜索过现有的 issues,确认这是一个新的问题与讨论。
Question Description / 问题描述

boot cmd:
when FPS_MAX_FRAMES = 16 is ok, but FPS_MAX_FRAMES=360 or larger has error.

PYTORCH_CUDA_ALLOC_CONF='expandable_segments:True' \
IMAGE_MAX_TOKEN_NUM=1024 \
VIDEO_MAX_TOKEN_NUM=256 \
FPS_MAX_FRAMES=360 \
NPROC_PER_NODE=2 \
CUDA_VISIBLE_DEVICES=0,1 \
swift sft \
    --model local_path/Qwen3-VL-8B-Instruct \
    --dataset 'dataset/data_train.jsonl' \
    --val_dataset 'dataset/data_val.jsonl' \
    --load_from_cache_file true \
    --split_dataset_ratio 0.01 \
    --train_type lora \
    --torch_dtype bfloat16 \
    --num_train_epochs 1 \
    --per_device_train_batch_size 1 \
    --per_device_eval_batch_size 1 \
    --attn_impl flash_attn \
    --padding_free true \
    --learning_rate 1e-4 \
    --lora_rank 8 \
    --lora_alpha 32 \
    --target_modules all-linear \
    --freeze_vit true \
    --freeze_aligner true \
    --packing true \
    --gradient_checkpointing true \
    --vit_gradient_checkpointing false \
    --gradient_accumulation_steps 2 \
    --eval_steps 100 \
    --save_steps 100 \
    --save_total_limit 2 \
    --logging_steps 5 \
    --max_length 4096 \
    --output_dir output \
    --warmup_ratio 0.05 \
    --deepspeed zero2 \
    --dataset_num_proc 4 \
    --dataloader_num_workers 4

Error log:

[rank0]: Traceback (most recent call last):
[rank0]:   File "/usr/local/miniconda3/lib/python3.10/site-packages/swift/cli/sft.py", line 20, in <module>
[rank0]:     sft_main()
[rank0]:   File "/usr/local/miniconda3/lib/python3.10/site-packages/swift/llm/train/sft.py", line 365, in sft_main
[rank0]:     return SwiftSft(args).main()
[rank0]:   File "/usr/local/miniconda3/lib/python3.10/site-packages/swift/llm/base.py", line 49, in main
[rank0]:     result = self.run()
[rank0]:   File "/usr/local/miniconda3/lib/python3.10/site-packages/swift/ray/base.py", line 170, in wrapper
[rank0]:     return func(self, *args, **kwargs)
[rank0]:   File "/usr/local/miniconda3/lib/python3.10/site-packages/swift/llm/train/sft.py", line 184, in run
[rank0]:     train_dataset, val_dataset = self._prepare_dataset()
[rank0]:   File "/usr/local/miniconda3/lib/python3.10/site-packages/swift/ray/base.py", line 170, in wrapper
[rank0]:     return func(self, *args, **kwargs)
[rank0]:   File "/usr/local/miniconda3/lib/python3.10/site-packages/swift/llm/train/sft.py", line 144, in _prepare_dataset
[rank0]:     datasets = self._post_process_datasets(datasets)
[rank0]:   File "/usr/local/miniconda3/lib/python3.10/site-packages/swift/llm/train/sft.py", line 160, in _post_process_datasets
[rank0]:     dataset = LazyLLMDataset(dataset, template.encode, strict=args.strict, random_state=args.data_seed)
[rank0]:   File "/usr/local/miniconda3/lib/python3.10/site-packages/swift/llm/dataset/utils.py", line 77, in __init__
[rank0]:     assert n_try_fetch >= 1
[rank0]: AssertionError

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with swift/llm/train/sft.py, following _prepare_dataset and _post_process_datasets into swift/llm/dataset/utils.py and LazyLLMDataset. Reproduce the command with FPS_MAX_FRAMES=16 and 360, then inspect why n_try_fetch reaches the assertion; done means the higher-frame configuration no longer fails during dataset preparation.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.