modelscope / modelscope/ms-swift
单标签分类任务微调时compute_acc函数发生报错
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 15.7k
- Forks
- 1.7k
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 136
Description
Checklist / 检查清单
- I have searched existing issues, and this is a new bug report. / 我已经搜索过现有的 issues,确认这是一个新的 bug report。
Bug Description / Bug 描述
[rank2]: Traceback (most recent call last):
[rank2]: File "/Multimodal/sujunyuan/ms_swift_sjy/ms-swift-main/swift/cli/sft.py", line 20, in
[rank2]: sft_main()
[rank2]: File "/Multimodal/sujunyuan/ms_swift_sjy/ms-swift-main/swift/pipelines/train/sft.py", line 354, in sft_main
[rank2]: return SwiftSft(args).main()
[rank2]: ^^^^^^^^^^^^^^^^^^^^^
[rank2]: File "/Multimodal/sujunyuan/ms_swift_sjy/ms-swift-main/swift/pipelines/base.py", line 52, in main
[rank2]: result = self.run()
[rank2]: ^^^^^^^^^^
[rank2]: File "/Multimodal/sujunyuan/ms_swift_sjy/ms-swift-main/swift/ray/base.py", line 168, in wrapper
[rank2]: return func(self, *args, **kwargs)
[rank2]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]: File "/Multimodal/sujunyuan/ms_swift_sjy/ms-swift-main/swift/pipelines/train/sft.py", line 197, in run
[rank2]: return self.train(trainer)
[rank2]: ^^^^^^^^^^^^^^^^^^^
[rank2]: File "/Multimodal/sujunyuan/ms_swift_sjy/ms-swift-main/swift/pipelines/train/sft.py", line 270, in train
[rank2]: trainer.train(resume_checkpoint)
[rank2]: File "/Multimodal/sujunyuan/ms_swift_sjy/ms-swift-main/swift/trainers/trainer.py", line 64, in train
[rank2]: return super().train(*args, **kwargs)
[rank2]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]: File "/Multimodal/sujunyuan/ms_swift_sjy/ms-swift-main/swift/trainers/mixin.py", line 916, in train
[rank2]: res = super().train(*args, **kwargs)
[rank2]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]: File "/usr/local/lib/python3.11/site-packages/transformers/trainer.py", line 2325, in train
[rank2]: return inner_training_loop(
[rank2]: ^^^^^^^^^^^^^^^^^^^^
[rank2]: File "/usr/local/lib/python3.11/site-packages/transformers/trainer.py", line 2674, in _inner_training_loop
[rank2]: tr_loss_step = self.training_step(model, inputs, num_items_in_batch)
[rank2]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]: File "/usr/local/lib/python3.11/site-packages/transformers/trainer.py", line 4020, in training_step
[rank2]: loss = self.compute_loss(model, inputs, num_items_in_batch=num_items_in_batch)
[rank2]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]: File "/Multimodal/sujunyuan/ms_swift_sjy/ms-swift-main/swift/trainers/trainer.py", line 69, in compute_loss
[rank2]: self._compute_acc(outputs, inputs['labels'])
[rank2]: File "/Multimodal/sujunyuan/ms_swift_sjy/ms-swift-main/swift/trainers/mixin.py", line 1037, in _compute_acc
[rank2]: metrics = compute_acc(preds, labels)
[rank2]: ^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]: File "/Multimodal/sujunyuan/ms_swift_sjy/ms-swift-main/swift/metrics/acc.py", line 20, in compute_acc
[rank2]: preds = preds.cpu().numpy()
[rank2]: ^^^^^^^^^^^
[rank2]: torch.AcceleratorError: CUDA error: device-side assert triggered
[rank2]: CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
[rank2]: For debugging consider passing CUDA_LAUNCH_BLOCKING=1
[rank2]: Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.
How to Reproduce / 如何复现
swift sft
--model '/Multimodal/Models/Qwen3-VL/Qwen3-VL-8B-Instruct'
--train_type full
--task_type seq_cls
--problem_type single_label_classification
--packing true
--num_labels 530
--save_strategy 'epoch'
--padding_free true
--dataset $traning_dataset
--load_from_cache_file false
--torch_dtype bfloat16
--enable_dft_loss false
--num_train_epochs 6
--per_device_train_batch_size 16
--learning_rate 1e-5
--gradient_accumulation_steps 2
--save_total_limit 6
--logging_steps 1
--max_length 1024
--output_dir $output_dir
--system ''
--warmup_ratio 0.1
--deepspeed zero1
--dataloader_num_workers 64
--report_to wandb
--attn_impl flash_attn
--dataset_num_proc 8 \
Additional Information / 补充信息
数据集格式为:{"messages": [{"role": "user", "content": ""}], "images": ["img.jpg"], "label": 0}
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with swift/metrics/acc.py and swift/trainers/mixin.py, following the _compute_acc call shown in the traceback. Re-run the provided single-label classification command with CUDA_LAUNCH_BLOCKING=1 to identify where the device-side assert originates. Done means training completes and accuracy is computed without the CUDA error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 42/100