modelscope / modelscope/ms-swift
RuntimeError: DataLoader worker is killed by signal: Bus error
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 15.7k
- Forks
- 1.7k
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 136
Description
Describe the bug
GRPO 训练过程中出现报错 RuntimeError: DataLoader worker is killed by signal: Bus error (具体错误信息如下图)
出现的比较随机,会在训练过程中概率出现,无法稳定复现,但遇到不止一次了。
查看系统共享内存大小:
df -h /dev/shm
Filesystem Size Used Avail Use% Mounted on
tmpfs 492G 0 492G 0% /dev/shm
Your hardware and system info
H20 96G * 8 (CUDA 12.8)
swift: 3.12.0.dev0
torch: 2.8.0
transformers: 4.57.3
Additional context
脚本
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \ MASTER_PORT=29601 \ NPROC_PER_NODE=8 \ swift rlhf \ --rlhf_type grpo \ --loss_type bnpo \ --model Qwen/Qwen2.5-7B \ --reward_funcs accuracy \ --use_vllm true \ --vllm_mode colocate \ --train_type full \ --torch_dtype bfloat16 \ --attn_impl flash_attention_2 \ --dataset xxx/DAPO-Math-17k-Processed/en.jsonl \ --split_dataset_ratio 0 \ --max_completion_length 2048 \ --num_train_epochs 1 \ --learning_rate 1e-6 \ --lr_scheduler_type constant \ --per_device_train_batch_size 8 \ --per_device_eval_batch_size 1 \ --gradient_accumulation_steps 16 \ --num_generations 8 \ --epsilon 0.2 \ --epsilon_high 0.2 \ --save_steps 10 \ --logging_steps 1 \ --warmup_ratio 0.0 \ --dataloader_num_workers 4 \ --dataset_num_proc 4 \ --vllm_gpu_memory_utilization 0.7 \ --temperature 0.8 \ --top_p 0.95 \ --top_k 50 \ --num_iterations 4 \ --deepspeed zero3_offload \ --output_dir ./checkpoints/stage2 \ --report_to swanlab \ --beta 0.001 \ --log_completions true \ --log_entropy true \ --sleep_level 1 \ --offload_model true
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the supplied swift rlhf command, especially --dataloader_num_workers 4, and compare the intermittent DataLoader worker failure with the reported /dev/shm information. The issue names no source file or test; done should mean identifying a reliable cause or reproduction and documenting a confirmed fix or mitigation for this configuration.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, swift
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100