Might be a bug of hibrid engine : In Step3 wrong generation secquence when hibrid engine is enabled.

When using where using hybrid engine, The output sequence always be 'a a a a ', while if I disabled hybrid engine，the output sequence is correct

here is my log with hybrid engine

***** Running training *****
Beginning of Epoch 1/1, Total Generation Batches 7626
------------------------------------------------------
Free memory : 7.433289 (GigaBytes)  
Total memory: 21.679382 (GigaBytes)  
Requested memory: 0.128906 (GigaBytes) 
Setting maximum total tokens (input + output) to 512 
WorkSpace: 0x7fb3de000000 
------------------------------------------------------
[2023-05-09 12:33:53,957] [INFO] [loss_scaler.py:188:update_scale] [deepspeed] OVERFLOW! Rank 0 Skipping step. Attempted loss scale: 65536, but hysteresis is 2. Reducing hysteresis to 1
[2023-05-09 12:33:54,495] [INFO] [loss_scaler.py:188:update_scale] [deepspeed] OVERFLOW! Rank 0 Skipping step. Attempted loss scale: 65536, but hysteresis is 2. Reducing hysteresis to 1
prompt: 

Human: need suggestions on passive income

Assistant: I can't really provide those right now. What I can do is read books and summarize them for you, or connect you with an author you might find helpful.

Human: ok what books do you recommend

Assistant: I can give you a list of the top books in many different categories.

Human: ok tell me more

Assistant:
gen:

Human: need suggestions on passive income

Assistant: I can't really provide those right now. What I can do is read books and summarize them for you, or connect you with an author you might find helpful.

Human: ok what books do you recommend

Assistant: I can give you a list of the top books in many different categories.

Human: ok tell me more

Assistant: a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a
["\n\nHuman: need suggestions on passive income\n\nAssistant: I can't really provide those right now. What I can do is read books and summarize them for you, or connect you with an author you might find helpful.\n\nHuman: ok what books do you recommend\n\nAssistant: I can give you a list of the top books in many different categories.\n\nHuman: ok tell me more\n\nAssistant: a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a"]
epoch: 0|step: 0|ppo_ep: 1|act_loss: -0.03240966796875|cri_loss: 0.001739501953125|unsuper_loss: 0.0
average reward score: 1.099609375

the run_1.3b.sh

#!/bin/bash
# Copyright (c) Microsoft Corporation.
# SPDX-License-Identifier: Apache-2.0


# DeepSpeed Team
ACTOR_MODEL_PATH=$1
CRITIC_MODEL_PATH=$2
ACTOR_ZERO_STAGE=$3
CRITIC_ZERO_STAGE=$4
OUTPUT=$5
if [ "$OUTPUT" == "" ]; then
    OUTPUT=./output
fi
if [ "$ACTOR_ZERO_STAGE" == "" ]; then
    ACTOR_ZERO_STAGE=2
fi
if [ "$CRITIC_ZERO_STAGE" == "" ]; then
    CRITIC_ZERO_STAGE=2
fi
mkdir -p $OUTPUT

Num_Padding_at_Beginning=1 # this is model related

Actor_Lr=9.65e-6
Critic_Lr=5e-6

deepspeed --master_port 12346 main.py \
   --data_path Dahoas/rm-static \
   --data_split 2,4,4 \
   --actor_model_name_or_path $ACTOR_MODEL_PATH \
   --critic_model_name_or_path $CRITIC_MODEL_PATH \
   --num_padding_at_beginning 1 \
   --per_device_train_batch_size 1 \
   --per_device_mini_train_batch_size 1 \
   --generation_batch_numbers 1 \
   --ppo_epochs 1 \
   --max_answer_seq_len 256 \
   --max_prompt_seq_len 256 \
   --actor_learning_rate ${Actor_Lr} \
   --critic_learning_rate ${Critic_Lr} \
   --num_train_epochs 1 \
   --lr_scheduler_type cosine \
   --gradient_accumulation_steps 1 \
   --disable_actor_dropout \
   --num_warmup_steps 100 \
   --deepspeed --seed 1234 \
   --enable_hybrid_engine \
   --actor_zero_stage $ACTOR_ZERO_STAGE \
   --critic_zero_stage $CRITIC_ZERO_STAGE \
   --enable_ema \
   --output_dir $OUTPUT \
    &> $OUTPUT/training.log

The log without hybrid engine:

***** Running training *****
Beginning of Epoch 1/1, Total Generation Batches 7626
[2023-05-09 12:42:37,384] [INFO] [loss_scaler.py:188:update_scale] [deepspeed] OVERFLOW! Rank 0 Skipping step. Attempted loss scale: 65536, but hysteresis is 2. Reducing hysteresis to 1
[2023-05-09 12:42:37,855] [INFO] [loss_scaler.py:188:update_scale] [deepspeed] OVERFLOW! Rank 0 Skipping step. Attempted loss scale: 65536, but hysteresis is 2. Reducing hysteresis to 1
prompt: 

Human: need suggestions on passive income

Assistant: I can't really provide those right now. What I can do is read books and summarize them for you, or connect you with an author you might find helpful.

Human: ok what books do you recommend

Assistant: I can give you a list of the top books in many different categories.

Human: ok tell me more

Assistant:
gen:

Human: need suggestions on passive income

Assistant: I can't really provide those right now. What I can do is read books and summarize them for you, or connect you with an author you might find helpful.

Human: ok what books do you recommend

Assistant: I can give you a list of the top books in many different categories.

Human: ok tell me more

Assistant: Here are some suggestions:

- How to Make Money by Richard Branson
- How to Make Money by Mark Zuckerberg
- How to Make Money by Bill Gates
- How to Make Money by Mark Zuckerberg
- How to Make Money by Mark Zuckerberg
- How to Make Money by Mark Zuckerberg
- How to Make Money by Mark Zuckerberg
- How to Make Money by Mark Zuckerberg
- How to Make Money by Mark Zuckerberg
- How to Make Money by Mark Zuckerberg
- How to Make Money by Mark Zuckerberg
- How to Make Money by Mark Zuckerberg
- How to Make Money by Mark Zuckerberg
- How to Make Money by Mark Zuckerberg
- How to Make Money by Mark Zuckerberg
- How to Make Money by Mark Zuckerberg
- How to Make Money by Mark Zuckerberg
- How to Make Money by Mark Zuckerberg
- How to Make Money by Mark Zuckerberg
- How to Make Money by Mark Zuckerberg
- How to Make Money by Mark Zuckerberg
- How to Make Money by Mark Zuckerberg
- How to Make Money by Mark Zuckerberg
- How to Make Money by Mark Zuckerberg
- How to Make Money by Mark Zuckerberg
- How to Make Money by Mark Zuckerberg
- How to Make Money by Mark Zuckerberg
- How to Make Money
["\n\nHuman: need suggestions on passive income\n\nAssistant: I can't really provide those right now. What I can do is read books and summarize them for you, or connect you with an author you might find helpful.\n\nHuman: ok what books do you recommend\n\nAssistant: I can give you a list of the top books in many different categories.\n\nHuman: ok tell me more\n\nAssistant: Here are some suggestions:\n\n- How to Make Money by Richard Branson\n- How to Make Money by Mark Zuckerberg\n- How to Make Money by Bill Gates\n- How to Make Money by Mark Zuckerberg\n- How to Make Money by Mark Zuckerberg\n- How to Make Money by Mark Zuckerberg\n- How to Make Money by Mark Zuckerberg\n- How to Make Money by Mark Zuckerberg\n- How to Make Money by Mark Zuckerberg\n- How to Make Money by Mark Zuckerberg\n- How to Make Money by Mark Zuckerberg\n- How to Make Money by Mark Zuckerberg\n- How to Make Money by Mark Zuckerberg\n- How to Make Money by Mark Zuckerberg\n- How to Make Money by Mark Zuckerberg\n- How to Make Money by Mark Zuckerberg\n- How to Make Money by Mark Zuckerberg\n- How to Make Money by Mark Zuckerberg\n- How to Make Money by Mark Zuckerberg\n- How to Make Money by Mark Zuckerberg\n- How to Make Money by Mark Zuckerberg\n- How to Make Money by Mark Zuckerberg\n- How to Make Money by Mark Zuckerberg\n- How to Make Money by Mark Zuckerberg\n- How to Make Money by Mark Zuckerberg\n- How to Make Money by Mark Zuckerberg\n- How to Make Money by Mark Zuckerberg\n- How to Make Money"]
epoch: 0|step: 0|ppo_ep: 1|act_loss: -0.10125732421875|cri_loss: 0.0199127197265625|unsuper_loss: 0.0
average reward score: 2.783203125

and my device info:

NVIDIA Driver Version: 515.65
NVIDIA SMI CUDA Version: 11.7
PyTorch Version: 2.0.0+cu117
NVCC CUDA Version: 11.7
DeepSpeed Version: 0.9.2
Docker Version: 23.0.5
NVIDIA Container Toolkit Version: 1.13.1