deepspeedai / deepspeedai/DeepSpeedExamples

program stop when i run my run_chinese.sh in GPU A100*4 80G

Open
#487 3 comments 0 reactions 1 assignee View on GitHub

@samadejacobs is already working on this.

Since May 18, 2023.

deespeed chat
Dominant language
Python
Stars
6.8k
Forks
1.1k
Avg merge
2d 16h
Merged PRs (30d)
1

Description

questions

hi guys,when i use A100 run run_chinese.sh example,i find the program stopped somewhere.But i run through English example.And program have not error reported.If this is a bug?Can you give me some advices?

train.log

L2hvbWUvbHl5Ly5jb25maWcvRGluZ1RhbGsvNzE4Njg2MzAxX3YyL0ltYWdlRmlsZXMvMTY4MzM3Mjc1OTAwMV8xNjgzMzcyNzU3NDMxLmpwZw==

my run_chinese.sh

#!/bin/bash
# Copyright (c) Microsoft Corporation.
# SPDX-License-Identifier: Apache-2.0
# DeepSpeed Team
OUTPUT=$1
ZERO_STAGE=$2
if [ "$OUTPUT" == "" ]; then
    OUTPUT=./output
fi
if [ "$ZERO_STAGE" == "" ]; then
    ZERO_STAGE=3
fi
mkdir -p $OUTPUT

# The Chinese data we found mostly only contain one response without another
# "rejected" response. Thus we only test the step 1 finetuning and use
# a data_split of 10,0,0 (keep all data for step 1).
deepspeed main.py \
   --data_path wangrui6/Zhihu-KOL \
   --data_split 10,0,0 \
   --model_name_or_path facebook/opt-1.3b \
   --per_device_train_batch_size 16 \
   --per_device_eval_batch_size 16 \
   --max_seq_len 512 \
   --learning_rate 9.65e-6 \
   --weight_decay 0. \
   --num_train_epochs 16 \
   --gradient_accumulation_steps 1 \
   --lr_scheduler_type cosine \
   --num_warmup_steps 0 \
   --seed 1234 \
   --zero_stage $ZERO_STAGE \
   --deepspeed \
   --output_dir $OUTPUT \
   &> $OUTPUT/training.log

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.