[Question] Step3, fatal error: cuda_bf16.h: No such file or directory

I trained actor model and critic model step by step and saved them in ./step1_supervised_finetuning/output and ./step2_reward_model_finetuning/output separately

Then I continue to run the third step by

bash training_scripts/single_gpu/run_1.3b.sh ./step1_supervised_finetuning/output ./step2_reward_model_finetuning/output

It shows a long error and I think the main error is on: ./inference_cublas_wrappers.h:12:10: fatal error: cuda_bf16.h: No such file or directory
#include <cuda_bf16.h>
^~~~~~~~~~~~~
compilation terminated.
My cuda version is 10.2 and I tried to download cuda_bf16.h/cuda_bf16.hpp into /usr/local/cuda-10.2/targets/x86_64-linux/include but still showsNo declaration matches '__half ::__half(__nv_bfloat16)'

I found the main error on the function create_inference_containers of class DeepSpeedHybridEngine so I change it to class DeepSpeedEngine by removing --enable_hybrid_engine in run_1.3b.sh

However, it shows error: cuda out of memory, even though I change all train_batch to 1 and set os.environ["PYTORCH_CUDA_ALLOC_CONF"] = "max_split_size_mb:32"

Start with training_scripts/single_gpu/run_1.3b.sh and trace the create_inference_containers call in DeepSpeedHybridEngine. Reproduce Step 3 with the reported CUDA 10.2 environment, then check the cuda_bf16.h compilation failure and the later out-of-memory error after disabling hybrid engine. Done means identifying a supported configuration or a documented fix that lets Step 3 run.