deepspeedai / deepspeedai/DeepSpeed
[BUG]I trained 1.3b model on V100 and reported an error
@jeffra is already working on this.
Since Apr 18, 2023.
- Dominant language
- Python
- Stars
- 43.1k
- Forks
- 5k
- Avg merge
- 4d 15h
- Merged PRs (30d)
- 112
Description
python train.py --actor-model facebook/opt-1.3b --reward-model facebook/opt-350m --deployment-type single_node/
python train.py --actor-model facebook/opt-1.3b --reward-model facebook/opt-350m --deployment-type single_gpu
[2023-04-15 08:26:00,653] [INFO] [launch.py:428:sigkill_handler] Killing subprocess 759
[2023-04-15 08:26:00,655] [ERROR] [launch.py:434:sigkill_handler] ['/mnt/data/virtualenv/venv/bin/python', '-u', 'main.py', '--local_rank=0', '--data_path', 'Dahoas/rm-static', 'Dahoas/full-hh-rlhf', 'Dahoas/synthetic-instruct-gptj-pairwise', 'yitingxie/rlhf-reward-datasets', 'openai/webgpt_comparisons', 'stanfordnlp/SHP', '--data_split', '2,4,4', '--model_name_or_path', 'facebook/opt-1.3b', '--per_device_train_batch_size', '8', '--per_device_eval_batch_size', '8', '--max_seq_len', '512', '--learning_rate', '9.65e-6', '--weight_decay', '0.1', '--num_train_epochs', '2', '--gradient_accumulation_steps', '1', '--lr_scheduler_type', 'cosine', '--num_warmup_steps', '0', '--seed', '1234', '--zero_stage', '2', '--deepspeed', '--output_dir', '/mnt/data/virtualenv/DeepSpeedExamples/applications/DeepSpeed-Chat/output/actor-models/1.3b'] exits with return code = -9
(venv) /mnt/data/virtualenv/DeepSpeedExamples/applications/DeepSpeed-Chat> python train.py --actor-model facebook/opt-2.7b --reward-model facebook/opt-350m --deployment-type single_node
usage: train.py [-h] [--step {1,2,3} [{1,2,3} ...]] [--actor-model {1.3b,6.7b,13b,66b}] [--reward-model {3,5,0,m}] [--actor-zero-stage {,0,1,2,3}] [--reward-zero-stage {,0,1,2,3}] [--output-dir OUTPUT_DIR]
[--deployment-type {single_gpu,single_node,multi_node}]
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.