deepspeedai / deepspeedai/DeepSpeed

[BUG]I trained 1.3b model on V100 and reported an error

Open
#3,244 9 comments 0 reactions 1 assignee View on GitHub

@jeffra is already working on this.

Since Apr 18, 2023.

bug deepspeed-chat
Dominant language
Python
Stars
43.1k
Forks
5k
Avg merge
4d 15h
Merged PRs (30d)
112

Description

python train.py --actor-model facebook/opt-1.3b --reward-model facebook/opt-350m --deployment-type single_node/
python train.py --actor-model facebook/opt-1.3b --reward-model facebook/opt-350m --deployment-type single_gpu

[2023-04-15 08:26:00,653] [INFO] [launch.py:428:sigkill_handler] Killing subprocess 759
[2023-04-15 08:26:00,655] [ERROR] [launch.py:434:sigkill_handler] ['/mnt/data/virtualenv/venv/bin/python', '-u', 'main.py', '--local_rank=0', '--data_path', 'Dahoas/rm-static', 'Dahoas/full-hh-rlhf', 'Dahoas/synthetic-instruct-gptj-pairwise', 'yitingxie/rlhf-reward-datasets', 'openai/webgpt_comparisons', 'stanfordnlp/SHP', '--data_split', '2,4,4', '--model_name_or_path', 'facebook/opt-1.3b', '--per_device_train_batch_size', '8', '--per_device_eval_batch_size', '8', '--max_seq_len', '512', '--learning_rate', '9.65e-6', '--weight_decay', '0.1', '--num_train_epochs', '2', '--gradient_accumulation_steps', '1', '--lr_scheduler_type', 'cosine', '--num_warmup_steps', '0', '--seed', '1234', '--zero_stage', '2', '--deepspeed', '--output_dir', '/mnt/data/virtualenv/DeepSpeedExamples/applications/DeepSpeed-Chat/output/actor-models/1.3b'] exits with return code = -9
(venv) /mnt/data/virtualenv/DeepSpeedExamples/applications/DeepSpeed-Chat> python train.py --actor-model facebook/opt-2.7b --reward-model facebook/opt-350m --deployment-type single_node
usage: train.py [-h] [--step {1,2,3} [{1,2,3} ...]] [--actor-model {1.3b,6.7b,13b,66b}] [--reward-model {3,5,0,m}] [--actor-zero-stage {,0,1,2,3}] [--reward-zero-stage {,0,1,2,3}] [--output-dir OUTPUT_DIR]
[--deployment-type {single_gpu,single_node,multi_node}]

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.