Fatal error: cuda_fp16.h: No such file or directory on ROCm

The following bugs occur when I try to run the example gpt-2 code:

```
[default0]:Building extension module scaled_upper_triang_masked_softmax_cuda...
[default0]:Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
[default0]:[1/3] c++ -MMD -MF scaled_upper_triang_masked_softmax.o.d -DTORCH_EXTENSION_NAME=scaled_upper_triang_masked_softmax_cuda -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -isystem /public/home/ach2ha8oau/miniconda3/envs/megatron-deepspeed-dtk/lib/python3.9/site-packages/torch/include -isystem /public/home/ach2ha8oau/miniconda3/envs/megatron-deepspeed-dtk/lib/python3.9/site-packages/torch/include/torch/csrc/api/include -isystem /public/home/ach2ha8oau/miniconda3/envs/megatron-deepspeed-dtk/lib/python3.9/site-packages/torch/include/TH -isystem /public/home/ach2ha8oau/miniconda3/envs/megatron-deepspeed-dtk/lib/python3.9/site-packages/torch/include/THC -isystem /public/home/ach2ha8oau/miniconda3/envs/megatron-deepspeed-dtk/lib/python3.9/site-packages/torch/include/THH -isystem /public/software/compiler/rocm/dtk-22.10/include -isystem /public/software/compiler/rocm/dtk-22.10/miopen/include -isystem /public/home/ach2ha8oau/miniconda3/envs/megatron-deepspeed-dtk/include/python3.9 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++14 -O3 -c /public/home/ach2ha8oau/megatron-deepspeed/Megatron-DeepSpeed-bigscience/megatron/fused_kernels/scaled_upper_triang_masked_softmax.cpp -o scaled_upper_triang_masked_softmax.o
[default0]:FAILED: scaled_upper_triang_masked_softmax.o
[default0]:c++ -MMD -MF scaled_upper_triang_masked_softmax.o.d -DTORCH_EXTENSION_NAME=scaled_upper_triang_masked_softmax_cuda -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -isystem /public/home/ach2ha8oau/miniconda3/envs/megatron-deepspeed-dtk/lib/python3.9/site-packages/torch/include -isystem /public/home/ach2ha8oau/miniconda3/envs/megatron-deepspeed-dtk/lib/python3.9/site-packages/torch/include/torch/csrc/api/include -isystem /public/home/ach2ha8oau/miniconda3/envs/megatron-deepspeed-dtk/lib/python3.9/site-packages/torch/include/TH -isystem /public/home/ach2ha8oau/miniconda3/envs/megatron-deepspeed-dtk/lib/python3.9/site-packages/torch/include/THC -isystem /public/home/ach2ha8oau/miniconda3/envs/megatron-deepspeed-dtk/lib/python3.9/site-packages/torch/include/THH -isystem /public/software/compiler/rocm/dtk-22.10/include -isystem /public/software/compiler/rocm/dtk-22.10/miopen/include -isystem /public/home/ach2ha8oau/miniconda3/envs/megatron-deepspeed-dtk/include/python3.9 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++14 -O3 -c /public/home/ach2ha8oau/megatron-deepspeed/Megatron-DeepSpeed-bigscience/megatron/fused_kernels/scaled_upper_triang_masked_softmax.cpp -o scaled_upper_triang_masked_softmax.o
[default0]:/public/home/ach2ha8oau/megatron-deepspeed/Megatron-DeepSpeed-bigscience/megatron/fused_kernels/scaled_upper_triang_masked_softmax.cpp:17:10: fatal error: cuda_fp16.h: No such file or directory
[default0]: #include
[default0]: ^~~~~~~~~~~~~
[default0]:compilation terminated.
[default0]:[2/3] /public/software/compiler/rocm/dtk-22.10/bin/hipcc -DWITH_HIP -DTORCH_EXTENSION_NAME=scaled_upper_triang_masked_softmax_cuda -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -isystem /public/home/ach2ha8oau/miniconda3/envs/megatron-deepspeed-dtk/lib/python3.9/site-packages/torch/include -isystem /public/home/ach2ha8oau/miniconda3/envs/megatron-deepspeed-dtk/lib/python3.9/site-packages/torch/include/torch/csrc/api/include -isystem /public/home/ach2ha8oau/miniconda3/envs/megatron-deepspeed-dtk/lib/python3.9/site-packages/torch/include/TH -isystem /public/home/ach2ha8oau/miniconda3/envs/megatron-deepspeed-dtk/lib/python3.9/site-packages/torch/include/THC -isystem /public/home/ach2ha8oau/miniconda3/envs/megatron-deepspeed-dtk/lib/python3.9/site-packages/torch/include/THH -isystem /public/software/compiler/rocm/dtk-22.10/include -isystem /public/software/compiler/rocm/dtk-22.10/miopen/include -isystem /public/home/ach2ha8oau/miniconda3/envs/megatron-deepspeed-dtk/include/python3.9 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++14 -O3 -fPIC -D__HIP_PLATFORM_HCC__=1 -DCUDA_HAS_FP16=1 -D__HIP_NO_HALF_OPERATORS__=1 -D__HIP_NO_HALF_CONVERSIONS__=1 -O3 --use_fast_math -U__CUDA_NO_HALF_OPERATORS__ -U__CUDA_NO_HALF_CONVERSIONS__ --expt-relaxed-constexpr --expt-extended-lambda --amdgpu-target=gfx900 --amdgpu-target=gfx906 -fno-gpu-rdc -c /public/home/ach2ha8oau/megatron-deepspeed/Megatron-DeepSpeed-bigscience/megatron/fused_kernels/scaled_upper_triang_masked_softmax_hip.hip -o scaled_upper_triang_masked_softmax_hip.cuda.o
[default0]:FAILED: scaled_upper_triang_masked_softmax_hip.cuda.o
[default0]:/public/software/compiler/rocm/dtk-22.10/bin/hipcc -DWITH_HIP -DTORCH_EXTENSION_NAME=scaled_upper_triang_masked_softmax_cuda -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -isystem /public/home/ach2ha8oau/miniconda3/envs/megatron-deepspeed-dtk/lib/python3.9/site-packages/torch/include -isystem /public/home/ach2ha8oau/miniconda3/envs/megatron-deepspeed-dtk/lib/python3.9/site-packages/torch/include/torch/csrc/api/include -isystem /public/home/ach2ha8oau/miniconda3/envs/megatron-deepspeed-dtk/lib/python3.9/site-packages/torch/include/TH -isystem /public/home/ach2ha8oau/miniconda3/envs/megatron-deepspeed-dtk/lib/python3.9/site-packages/torch/include/THC -isystem /public/home/ach2ha8oau/miniconda3/envs/megatron-deepspeed-dtk/lib/python3.9/site-packages/torch/include/THH -isystem /public/software/compiler/rocm/dtk-22.10/include -isystem /public/software/compiler/rocm/dtk-22.10/miopen/include -isystem /public/home/ach2ha8oau/miniconda3/envs/megatron-deepspeed-dtk/include/python3.9 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++14 -O3 -fPIC -D__HIP_PLATFORM_HCC__=1 -DCUDA_HAS_FP16=1 -D__HIP_NO_HALF_OPERATORS__=1 -D__HIP_NO_HALF_CONVERSIONS__=1 -O3 --use_fast_math -U__CUDA_NO_HALF_OPERATORS__ -U__CUDA_NO_HALF_CONVERSIONS__ --expt-relaxed-constexpr --expt-extended-lambda --amdgpu-target=gfx900 --amdgpu-target=gfx906 -fno-gpu-rdc -c /public/home/ach2ha8oau/megatron-deepspeed/Megatron-DeepSpeed-bigscience/megatron/fused_kernels/scaled_upper_triang_masked_softmax_hip.hip -o scaled_upper_triang_masked_softmax_hip.cuda.o
[default0]:clang-14: error: unsupported option '--use_fast_math'
[default0]:clang-14: error: unsupported option '--expt-relaxed-constexpr'
[default0]:clang-14: error: unsupported option '--expt-extended-lambda'
[default0]:ninja: build stopped: subcommand failed.
[default0]:Traceback (most recent call last):
[default0]: File "/public/home/ach2ha8oau/miniconda3/envs/megatron-deepspeed-dtk/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1718, in _run_ninja_build
[default0]: subprocess.run(
[default0]: File "/public/home/ach2ha8oau/miniconda3/envs/megatron-deepspeed-dtk/lib/python3.9/subprocess.py", line 528, in run
[default0]: raise CalledProcessError(retcode, process.args,
[default0]:subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.
```
It seems that it doesn't building extension module scaled_upper_triang_masked_softmax_cuda properly.

Inizia da megatron/fused_kernels/scaled_upper_triang_masked_softmax.cpp e scaled_upper_triang_masked_softmax_hip.hip, quindi esamina il comando di build dell’estensione C++ di PyTorch mostrato nel report. Riproduci l’esempio GPT-2 sulla configurazione ROCm indicata e determina perché entrambi i percorsi di compilazione falliscono. Il lavoro è completato quando l’estensione scaled_upper_triang_masked_softmax_cuda viene compilata e l’esempio viene eseguito senza questi errori.

Scritto dal modello di indicizzazione a partire dal testo della issue.