[BUG]I cannot run use DeepSpeed Chat train my model with enabled "hybrid_engine" in step3

Describe the bug
when i set "hybrid_engine" for making step3 training faster, the training progress is not stabilization, and often errors occur after just running or running a few steps

in step3:

1. Actor model and Critic model is 7b1 bloom
2. train dataset size is 4k

I mainly occur the following problems:

1. RuntimeError: inflight params error when using DeepSpeed for Reinforcement Learning
2. Illegal memory access CUDA error when using long sequences

when i cancel this parameter， I can run step3 code successfully, but is too slower

System info

• GPU count and types: one machines with x8 A100s each
• deepspeed version: 0.10.0

Bug info
!!!! kernel execution error. (m: 16384, n: 1, k: 4096, error: 13)
!!!! kernel execution error. (m: 4096, n: 1, k: 16384, error: 13)
!!!! kernel execution error. (m: 12288, n: 1, k: 4096, error: 13)
Traceback (most recent call last):
File "main.py", line 519, in
main()
File "main.py", line 425, in main
out = trainer.generate_experience(prompts)
File "/mnt/bn/simple-nas/mlx/users/zhangyawei.ywsq/RLHF_framework/DeepSpeed_Chat/DeepSpeedExamples/applications/DeepSpeed-Chat/training/step3_rlhf_finetuning/ppo_trainer.py", line 97, in generate_experience
seq = self._generate_sequence(prompts)
File "/mnt/bn/simple-nas/mlx/users/zhangyawei.ywsq/RLHF_framework/DeepSpeed_Chat/DeepSpeedExamples/applications/DeepSpeed-Chat/training/step3_rlhf_finetuning/ppo_trainer.py", line 75, in _generate_sequence
min_length=max_min_length)
File "/usr/local/lib/python3.7/dist-packages/deepspeed/runtime/hybrid_engine.py", line 266, in generate
generate_ret_vals = self._generate(*inputs, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/transformers/generation_utils.py", line 1328, in generate
**model_kwargs,
File "/usr/local/lib/python3.7/dist-packages/transformers/generation_utils.py", line 1717, in greedy_search
output_hidden_states=output_hidden_states,
File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/transformers/models/bloom/modeling_bloom.py", line 868, in forward
return_dict=return_dict,
File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/transformers/models/bloom/modeling_bloom.py", line 751, in forward
alibi=alibi,
File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/deepspeed/model_implementations/transformers/ds_transformer.py", line 181, in forward
alibi)
File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/deepspeed/ops/transformer/inference/ds_attention.py", line 161, in forward
alibi=alibi)
File "/usr/local/lib/python3.7/dist-packages/deepspeed/ops/transformer/inference/ds_attention.py", line 226, in compute_attention
query_layer, key_layer, value_layer = self._split_tensor_along_last_dim(mixed_x_layer, 3)
File "/usr/local/lib/python3.7/dist-packages/deepspeed/ops/transformer/inference/ds_attention.py", line 207, in _split_tensor_along_last_dim
return tuple(chunk.contiguous() for chunk in tensor_list)
File "/usr/local/lib/python3.7/dist-packages/deepspeed/ops/transformer/inference/ds_attention.py", line 207, in
return tuple(chunk.contiguous() for chunk in tensor_list)
RuntimeError: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.