deepspeedai / deepspeedai/DeepSpeed
[BUG]exits with return code = -9
@mrwyattii is already working on this.
Since Aug 21, 2023.
- Dominant language
- Python
- Stars
- 43.1k
- Forks
- 5k
- Avg merge
- 4d 15h
- Merged PRs (30d)
- 112
Description
Describe the bug
I can train with a single 3090, but with two 3090s I get an error with no prompting.
ds_report output
Screenshots
System info (please complete the following information):
- OS: [e.g. Ubuntu 18.04]
- GPU count and types [e.g. two machines with x8 A100s each]
- Interconnects (if applicable) [e.g., two machines connected with 100 Gbps IB]
- Python version
- Any other relevant info about your setup
Launcher context
deepspeed --num_gpus=2 fastchat/train/train_lora.py \
--model_name_or_path ../vicuna-13b-v1.5 \
--lora_r 8 \
--lora_alpha 16 \
--lora_dropout 0.05 \
--data_path ../4k_gossip_real_train.json \
--bf16 True \
--output_dir ../vicuna-13b-gossip-output \
--num_train_epochs 1 \
--per_device_train_batch_size 1 \
--per_device_eval_batch_size 1 \
--gradient_accumulation_steps 1 \
--evaluation_strategy "no" \
--save_strategy "steps" \
--save_steps 1200 \
--save_total_limit 100 \
--learning_rate 2e-5 \
--weight_decay 0. \
--warmup_ratio 0.03 \
--lr_scheduler_type "cosine" \
--logging_steps 1 \
--model_max_length 4100 \
--tf32 True \
--q_lora True \
--deepspeed playground/deepspeed_config_s2.json \
--gradient_checkpointing True
Docker context
I'm in an lxc container.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.