modelscope / modelscope/ms-swift
pretrain阶段,streaming 和 packing不兼容的问题
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 15.7k
- Forks
- 1.7k
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 136
Description
Checklist / 检查清单
- I have searched existing issues, and this is a new bug report. / 我已经搜索过现有的 issues,确认这是一个新的 bug report。
Bug Description / Bug 描述
Rank(Worker): 0 caught user code error, which triggered following unhandled exception:
Exception type: TypeError
Detail:
Traceback (most recent call last):
File "/checkpoint/binary/train_package/swift/cli/pt.py", line 7, in
pretrain_main()
File "/checkpoint/binary/train_package/swift/pipelines/train/pretrain.py", line 17, in pretrain_main
return SwiftPretrain(args).main()
File "/checkpoint/binary/train_package/swift/pipelines/base.py", line 52, in main
result = self.run()
File "/checkpoint/binary/train_package/swift/ray/base.py", line 168, in wrapper
return func(self, *args, **kwargs)
File "/checkpoint/binary/train_package/swift/pipelines/train/sft.py", line 174, in run
train_dataset, val_dataset = self._prepare_dataset()
File "/checkpoint/binary/train_package/swift/ray/base.py", line 168, in wrapper
return func(self, *args, **kwargs)
File "/checkpoint/binary/train_package/swift/pipelines/train/sft.py", line 135, in _prepare_dataset
self._show_dataset(*datasets)
File "/checkpoint/binary/train_package/swift/pipelines/train/sft.py", line 295, in _show_dataset
inputs = train_dataset[0] if hasattr(train_dataset, 'len') else next(iter(train_dataset))
File "/checkpoint/binary/train_package/swift/dataset/packing.py", line 196, in iter
data = self._fetch_data_out_queue(data, num_samples)
File "/checkpoint/binary/train_package/swift/dataset/packing.py", line 171, in _fetch_data_out_queue
res[i] = (data, len(data['input_ids']))
TypeError: list indices must be integers or slices, not str
让llm查了一下,发现是packing的时候需要input id,但是streaming的时候做不到这点,故而报错。
How to Reproduce / 如何复现
args="--model ${CKPT_PATH}
--max_steps 5000
--save_strategy steps
--shuffle_buffer_size 10000
--streaming true
--model_type qwen3
--train_type full
--dataset ${DATA_PATHS[@]}
--val_dataset ${VAL_DATA_PATHS[@]}
--packing true
--packing_num_proc ${num}
--enable_channel_loss true
--per_device_eval_batch_size 2
--eval_steps 50
--dataset_shuffle true
--torch_dtype bfloat16
--per_device_train_batch_size 2
--sequence_parallel_size 1
--learning_rate 1e-5
--gradient_accumulation_steps 2
--num_train_epochs 1
--truncation_strategy split
--max_length 8192
--save_steps 200
--logging_steps 1
--save_total_limit 100
--warmup_ratio 0.05
--dataloader_num_workers ${num}
--dataset_num_proc ${num}
--save_only_model true
--overwrite_output_dir true
--deepspeed zero3
--attn_impl flash_attention_2
--output_dir /data/oss_bucket_0/qingxi_hzx/cpt_model/${SCRIPT_NAME}/
--report_to swanlab
--run_name ${SCRIPT_NAME}
--swanlab_token ${SWANLAB_API_KEY}
--swanlab_project ${SWANLAB_PROJECT}"
Additional Information / 补充信息
No response
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with dataset/packing.py, especially iter and _fetch_data_out_queue, then inspect the dataset preparation and preview flow in pipelines/train/sft.py. Reproduce with streaming true and packing true using the provided pretraining arguments. Done means this combination no longer raises the reported TypeError and the dataset preview or training can proceed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100