modelscope / modelscope/ms-swift

RuntimeError: DataLoader worker is killed by signal: Bus error

Open
#7,264 4 comments 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
15.7k
Forks
1.7k
Avg merge
1d 16h
Merged PRs (30d)
136

Description

Describe the bug
GRPO 训练过程中出现报错 RuntimeError: DataLoader worker is killed by signal: Bus error (具体错误信息如下图)

Image

出现的比较随机,会在训练过程中概率出现,无法稳定复现,但遇到不止一次了。

查看系统共享内存大小:
df -h /dev/shm
Filesystem Size Used Avail Use% Mounted on
tmpfs 492G 0 492G 0% /dev/shm

Your hardware and system info
H20 96G * 8 (CUDA 12.8)
swift: 3.12.0.dev0
torch: 2.8.0
transformers: 4.57.3

Additional context
脚本
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \ MASTER_PORT=29601 \ NPROC_PER_NODE=8 \ swift rlhf \ --rlhf_type grpo \ --loss_type bnpo \ --model Qwen/Qwen2.5-7B \ --reward_funcs accuracy \ --use_vllm true \ --vllm_mode colocate \ --train_type full \ --torch_dtype bfloat16 \ --attn_impl flash_attention_2 \ --dataset xxx/DAPO-Math-17k-Processed/en.jsonl \ --split_dataset_ratio 0 \ --max_completion_length 2048 \ --num_train_epochs 1 \ --learning_rate 1e-6 \ --lr_scheduler_type constant \ --per_device_train_batch_size 8 \ --per_device_eval_batch_size 1 \ --gradient_accumulation_steps 16 \ --num_generations 8 \ --epsilon 0.2 \ --epsilon_high 0.2 \ --save_steps 10 \ --logging_steps 1 \ --warmup_ratio 0.0 \ --dataloader_num_workers 4 \ --dataset_num_proc 4 \ --vllm_gpu_memory_utilization 0.7 \ --temperature 0.8 \ --top_p 0.95 \ --top_k 50 \ --num_iterations 4 \ --deepspeed zero3_offload \ --output_dir ./checkpoints/stage2 \ --report_to swanlab \ --beta 0.001 \ --log_completions true \ --log_entropy true \ --sleep_level 1 \ --offload_model true

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the supplied swift rlhf command, especially --dataloader_num_workers 4, and compare the intermittent DataLoader worker failure with the reported /dev/shm information. The issue names no source file or test; done should mean identifying a reliable cause or reproduction and documenting a confirmed fix or mitigation for this configuration.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, swift
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.