deepspeedai / deepspeedai/DeepSpeed

[BUG]exits with return code = -9

Open
#4,181 5 comments 0 reactions 1 assignee View on GitHub

@mrwyattii is already working on this.

Since Aug 21, 2023.

bug training
Dominant language
Python
Stars
43.1k
Forks
5k
Avg merge
4d 15h
Merged PRs (30d)
112

Description

Describe the bug
I can train with a single 3090, but with two 3090s I get an error with no prompting.

ds_report output
image

Screenshots
image

System info (please complete the following information):

  • OS: [e.g. Ubuntu 18.04]
  • GPU count and types [e.g. two machines with x8 A100s each]
  • Interconnects (if applicable) [e.g., two machines connected with 100 Gbps IB]
  • Python version
  • Any other relevant info about your setup

Launcher context
deepspeed --num_gpus=2 fastchat/train/train_lora.py \ --model_name_or_path ../vicuna-13b-v1.5 \ --lora_r 8 \ --lora_alpha 16 \ --lora_dropout 0.05 \ --data_path ../4k_gossip_real_train.json \ --bf16 True \ --output_dir ../vicuna-13b-gossip-output \ --num_train_epochs 1 \ --per_device_train_batch_size 1 \ --per_device_eval_batch_size 1 \ --gradient_accumulation_steps 1 \ --evaluation_strategy "no" \ --save_strategy "steps" \ --save_steps 1200 \ --save_total_limit 100 \ --learning_rate 2e-5 \ --weight_decay 0. \ --warmup_ratio 0.03 \ --lr_scheduler_type "cosine" \ --logging_steps 1 \ --model_max_length 4100 \ --tf32 True \ --q_lora True \ --deepspeed playground/deepspeed_config_s2.json \ --gradient_checkpointing True

Docker context
I'm in an lxc container.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.