RuntimeError: Error building extension 'transformer_inference'

Using /root/.cache/torch_extensions/py310_cu117 as PyTorch extensions root...
Detected CUDA files, patching ldflags
Emitting ninja build file /root/.cache/torch_extensions/py310_cu117/transformer_inference/build.ninja...
Building extension module transformer_inference...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
[1/4] /usr/local/cuda/bin/nvcc -DTORCH_EXTENSION_NAME=transformer_inference -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="gcc" -DPYBIND11_STDLIB="libstdcpp" -DPYBIND11_BUILD_ABI="cxxabi1011" -I/data/v-yulinli/workspace/anaconda3/envs/deepspeed/lib/python3.10/site-packages/deepspeed/ops/csrc/transformer/inference/includes -I/data/v-yulinli/workspace/anaconda3/envs/deepspeed/lib/python3.10/site-packages/deepspeed/ops/csrc/includes -isystem /data/v-yulinli/workspace/anaconda3/envs/deepspeed/lib/python3.10/site-packages/torch/include -isystem /data/v-yulinli/workspace/anaconda3/envs/deepspeed/lib/python3.10/site-packages/torch/include/torch/csrc/api/include -isystem /data/v-yulinli/workspace/anaconda3/envs/deepspeed/lib/python3.10/site-packages/torch/include/TH -isystem /data/v-yulinli/workspace/anaconda3/envs/deepspeed/lib/python3.10/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /data/v-yulinli/workspace/anaconda3/envs/deepspeed/include/python3.10 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS_ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 --compiler-options '-fPIC' -std=c++17 -c /data/v-yulinli/workspace/anaconda3/envs/deepspeed/lib/python3.10/site-packages/deepspeed/ops/csrc/transformer/inference/csrc/layer_norm.cu -o layer_norm.cuda.o
FAILED: layer_norm.cuda.o
/usr/local/cuda/bin/nvcc -DTORCH_EXTENSION_NAME=transformer_inference -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="gcc" -DPYBIND11_STDLIB="libstdcpp" -DPYBIND11_BUILD_ABI="cxxabi1011" -I/data/v-yulinli/workspace/anaconda3/envs/deepspeed/lib/python3.10/site-packages/deepspeed/ops/csrc/transformer/inference/includes -I/data/v-yulinli/workspace/anaconda3/envs/deepspeed/lib/python3.10/site-packages/deepspeed/ops/csrc/includes -isystem /data/v-yulinli/workspace/anaconda3/envs/deepspeed/lib/python3.10/site-packages/torch/include -isystem /data/v-yulinli/workspace/anaconda3/envs/deepspeed/lib/python3.10/site-packages/torch/include/torch/csrc/api/include -isystem /data/v-yulinli/workspace/anaconda3/envs/deepspeed/lib/python3.10/site-packages/torch/include/TH -isystem /data/v-yulinli/workspace/anaconda3/envs/deepspeed/lib/python3.10/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /data/v-yulinli/workspace/anaconda3/envs/deepspeed/include/python3.10 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS_ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 --compiler-options '-fPIC' -std=c++17 -c /data/v-yulinli/workspace/anaconda3/envs/deepspeed/lib/python3.10/site-packages/deepspeed/ops/csrc/transformer/inference/csrc/layer_norm.cu -o layer_norm.cuda.o
/data/v-yulinli/workspace/anaconda3/envs/deepspeed/lib/python3.10/site-packages/deepspeed/ops/csrc/includes/reduction_utils.h(171): error: no operator "+" matches these operands
operand types are: const __half + const __half
return lhs + rhs;
^

/data/v-yulinli/workspace/anaconda3/envs/deepspeed/lib/python3.10/site-packages/deepspeed/ops/csrc/includes/reduction_utils.h(181): error: no operator ">" matches these operands
operand types are: const __half > const __half
return (lhs > rhs) ? lhs : rhs;
^

/data/v-yulinli/workspace/anaconda3/envs/deepspeed/lib/python3.10/site-packages/deepspeed/ops/csrc/includes/reduction_utils.h(192): error: no operator "<" matches these operands
operand types are: const __half < const __half
return (lhs < rhs) ? lhs : rhs;
^

/data/v-yulinli/workspace/anaconda3/envs/deepspeed/lib/python3.10/site-packages/deepspeed/ops/csrc/includes/reduction_utils.h(200): error: no operator "+" matches these operands
operand types are: const __half2 + const __half2
return lhs + rhs;
^

/data/v-yulinli/workspace/anaconda3/envs/deepspeed/lib/python3.10/site-packages/deepspeed/ops/csrc/includes/reduction_utils.h(210): error: no operator ">" matches these operands
operand types are: const __half > const __half
ret_val.x = (lhs.x > rhs.x) ? lhs.x : rhs.x;
^

/data/v-yulinli/workspace/anaconda3/envs/deepspeed/lib/python3.10/site-packages/deepspeed/ops/csrc/includes/reduction_utils.h(211): error: no operator ">" matches these operands
operand types are: const __half > const __half
ret_val.y = (lhs.y > rhs.y) ? lhs.y : rhs.y;
^

/data/v-yulinli/workspace/anaconda3/envs/deepspeed/lib/python3.10/site-packages/deepspeed/ops/csrc/includes/reduction_utils.h(223): error: no operator "<" matches these operands
operand types are: const __half < const __half
ret_val.x = (lhs.x < rhs.x) ? lhs.x : rhs.x;
^

/data/v-yulinli/workspace/anaconda3/envs/deepspeed/lib/python3.10/site-packages/deepspeed/ops/csrc/includes/reduction_utils.h(224): error: no operator "<" matches these operands
operand types are: const __half < const __half
ret_val.y = (lhs.y < rhs.y) ? lhs.y : rhs.y;
^

8 errors detected in the compilation of "/data/v-yulinli/workspace/anaconda3/envs/deepspeed/lib/python3.10/site-packages/deepspeed/ops/csrc/transformer/inference/csrc/layer_norm.cu".
[2/4] /usr/local/cuda/bin/nvcc -DTORCH_EXTENSION_NAME=transformer_inference -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="gcc" -DPYBIND11_STDLIB="libstdcpp" -DPYBIND11_BUILD_ABI="cxxabi1011" -I/data/v-yulinli/workspace/anaconda3/envs/deepspeed/lib/python3.10/site-packages/deepspeed/ops/csrc/transformer/inference/includes -I/data/v-yulinli/workspace/anaconda3/envs/deepspeed/lib/python3.10/site-packages/deepspeed/ops/csrc/includes -isystem /data/v-yulinli/workspace/anaconda3/envs/deepspeed/lib/python3.10/site-packages/torch/include -isystem /data/v-yulinli/workspace/anaconda3/envs/deepspeed/lib/python3.10/site-packages/torch/include/torch/csrc/api/include -isystem /data/v-yulinli/workspace/anaconda3/envs/deepspeed/lib/python3.10/site-packages/torch/include/TH -isystem /data/v-yulinli/workspace/anaconda3/envs/deepspeed/lib/python3.10/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /data/v-yulinli/workspace/anaconda3/envs/deepspeed/include/python3.10 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS_ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 --compiler-options '-fPIC' -std=c++17 -c /data/v-yulinli/workspace/anaconda3/envs/deepspeed/lib/python3.10/site-packages/deepspeed/ops/csrc/transformer/inference/csrc/rms_norm.cu -o rms_norm.cuda.o
FAILED: rms_norm.cuda.o
/usr/local/cuda/bin/nvcc -DTORCH_EXTENSION_NAME=transformer_inference -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="gcc" -DPYBIND11_STDLIB="libstdcpp" -DPYBIND11_BUILD_ABI="cxxabi1011" -I/data/v-yulinli/workspace/anaconda3/envs/deepspeed/lib/python3.10/site-packages/deepspeed/ops/csrc/transformer/inference/includes -I/data/v-yulinli/workspace/anaconda3/envs/deepspeed/lib/python3.10/site-packages/deepspeed/ops/csrc/includes -isystem /data/v-yulinli/workspace/anaconda3/envs/deepspeed/lib/python3.10/site-packages/torch/include -isystem /data/v-yulinli/workspace/anaconda3/envs/deepspeed/lib/python3.10/site-packages/torch/include/torch/csrc/api/include -isystem /data/v-yulinli/workspace/anaconda3/envs/deepspeed/lib/python3.10/site-packages/torch/include/TH -isystem /data/v-yulinli/workspace/anaconda3/envs/deepspeed/lib/python3.10/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /data/v-yulinli/workspace/anaconda3/envs/deepspeed/include/python3.10 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS_ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 --compiler-options '-fPIC' -std=c++17 -c /data/v-yulinli/workspace/anaconda3/envs/deepspeed/lib/python3.10/site-packages/deepspeed/ops/csrc/transformer/inference/csrc/rms_norm.cu -o rms_norm.cuda.o
/data/v-yulinli/workspace/anaconda3/envs/deepspeed/lib/python3.10/site-packages/deepspeed/ops/csrc/includes/reduction_utils.h(171): error: no operator "+" matches these operands
operand types are: const __half + const __half
return lhs + rhs;
^

/data/v-yulinli/workspace/anaconda3/envs/deepspeed/lib/python3.10/site-packages/deepspeed/ops/csrc/includes/reduction_utils.h(181): error: no operator ">" matches these operands
operand types are: const __half > const __half
return (lhs > rhs) ? lhs : rhs;
^

/data/v-yulinli/workspace/anaconda3/envs/deepspeed/lib/python3.10/site-packages/deepspeed/ops/csrc/includes/reduction_utils.h(192): error: no operator "<" matches these operands
operand types are: const __half < const __half
return (lhs < rhs) ? lhs : rhs;
^

/data/v-yulinli/workspace/anaconda3/envs/deepspeed/lib/python3.10/site-packages/deepspeed/ops/csrc/includes/reduction_utils.h(200): error: no operator "+" matches these operands
operand types are: const __half2 + const __half2
return lhs + rhs;
^

/data/v-yulinli/workspace/anaconda3/envs/deepspeed/lib/python3.10/site-packages/deepspeed/ops/csrc/includes/reduction_utils.h(210): error: no operator ">" matches these operands
operand types are: const __half > const __half
ret_val.x = (lhs.x > rhs.x) ? lhs.x : rhs.x;
^

/data/v-yulinli/workspace/anaconda3/envs/deepspeed/lib/python3.10/site-packages/deepspeed/ops/csrc/includes/reduction_utils.h(211): error: no operator ">" matches these operands
operand types are: const __half > const __half
ret_val.y = (lhs.y > rhs.y) ? lhs.y : rhs.y;
^

/data/v-yulinli/workspace/anaconda3/envs/deepspeed/lib/python3.10/site-packages/deepspeed/ops/csrc/includes/reduction_utils.h(223): error: no operator "<" matches these operands
operand types are: const __half < const __half
ret_val.x = (lhs.x < rhs.x) ? lhs.x : rhs.x;
^

/data/v-yulinli/workspace/anaconda3/envs/deepspeed/lib/python3.10/site-packages/deepspeed/ops/csrc/includes/reduction_utils.h(224): error: no operator "<" matches these operands
operand types are: const __half < const __half
ret_val.y = (lhs.y < rhs.y) ? lhs.y : rhs.y;
^

8 errors detected in the compilation of "/data/v-yulinli/workspace/anaconda3/envs/deepspeed/lib/python3.10/site-packages/deepspeed/ops/csrc/transformer/inference/csrc/rms_norm.cu".
[3/4] /usr/local/cuda/bin/nvcc -DTORCH_EXTENSION_NAME=transformer_inference -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="gcc" -DPYBIND11_STDLIB="libstdcpp" -DPYBIND11_BUILD_ABI="cxxabi1011" -I/data/v-yulinli/workspace/anaconda3/envs/deepspeed/lib/python3.10/site-packages/deepspeed/ops/csrc/transformer/inference/includes -I/data/v-yulinli/workspace/anaconda3/envs/deepspeed/lib/python3.10/site-packages/deepspeed/ops/csrc/includes -isystem /data/v-yulinli/workspace/anaconda3/envs/deepspeed/lib/python3.10/site-packages/torch/include -isystem /data/v-yulinli/workspace/anaconda3/envs/deepspeed/lib/python3.10/site-packages/torch/include/torch/csrc/api/include -isystem /data/v-yulinli/workspace/anaconda3/envs/deepspeed/lib/python3.10/site-packages/torch/include/TH -isystem /data/v-yulinli/workspace/anaconda3/envs/deepspeed/lib/python3.10/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /data/v-yulinli/workspace/anaconda3/envs/deepspeed/include/python3.10 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS_ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 --compiler-options '-fPIC' -std=c++17 -c /data/v-yulinli/workspace/anaconda3/envs/deepspeed/lib/python3.10/site-packages/deepspeed/ops/csrc/transformer/inference/csrc/gelu.cu -o gelu.cuda.o
FAILED: gelu.cuda.o
/usr/local/cuda/bin/nvcc -DTORCH_EXTENSION_NAME=transformer_inference -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="gcc" -DPYBIND11_STDLIB="libstdcpp" -DPYBIND11_BUILD_ABI="cxxabi1011" -I/data/v-yulinli/workspace/anaconda3/envs/deepspeed/lib/python3.10/site-packages/deepspeed/ops/csrc/transformer/inference/includes -I/data/v-yulinli/workspace/anaconda3/envs/deepspeed/lib/python3.10/site-packages/deepspeed/ops/csrc/includes -isystem /data/v-yulinli/workspace/anaconda3/envs/deepspeed/lib/python3.10/site-packages/torch/include -isystem /data/v-yulinli/workspace/anaconda3/envs/deepspeed/lib/python3.10/site-packages/torch/include/torch/csrc/api/include -isystem /data/v-yulinli/workspace/anaconda3/envs/deepspeed/lib/python3.10/site-packages/torch/include/TH -isystem /data/v-yulinli/workspace/anaconda3/envs/deepspeed/lib/python3.10/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /data/v-yulinli/workspace/anaconda3/envs/deepspeed/include/python3.10 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS_ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 --compiler-options '-fPIC' -std=c++17 -c /data/v-yulinli/workspace/anaconda3/envs/deepspeed/lib/python3.10/site-packages/deepspeed/ops/csrc/transformer/inference/csrc/gelu.cu -o gelu.cuda.o
/data/v-yulinli/workspace/anaconda3/envs/deepspeed/lib/python3.10/site-packages/deepspeed/ops/csrc/transformer/inference/csrc/gelu.cu(451): error: no operator "*" matches these operands
operand types are: __half * __half
mlp[idx] = mlp[idx] * coef2[idx] + res[idx] * coef1[idx];
^
detected during:
instantiation of "void moe_res_matmul(T *, T *, T *, int, int) [with T=__half]" at line 469
instantiation of "void launch_moe_res_matmul(T *, T *, T *, int, int, cudaStream_t) [with T=__half]" at line 479

/data/v-yulinli/workspace/anaconda3/envs/deepspeed/lib/python3.10/site-packages/deepspeed/ops/csrc/transformer/inference/csrc/gelu.cu(451): error: no operator "*" matches these operands
operand types are: __half * __half
mlp[idx] = mlp[idx] * coef2[idx] + res[idx] * coef1[idx];
^
detected during:
instantiation of "void moe_res_matmul(T *, T *, T *, int, int) [with T=__half]" at line 469
instantiation of "void launch_moe_res_matmul(T *, T *, T *, int, int, cudaStream_t) [with T=__half]" at line 479

/data/v-yulinli/workspace/anaconda3/envs/deepspeed/lib/python3.10/site-packages/deepspeed/ops/csrc/transformer/inference/csrc/gelu.cu(651): error: no operator "+" matches these operands
operand types are: __half + __half
T hidden_state = activation_buffer_1[v] + bias_buffer_1[v];
^
detected during:
instantiation of "void fused_gate_activation<T,useGelu>(T *, const T *, const T *, int, int, int) [with T=__half, useGelu=true]" at line 695
instantiation of "void launch_gated_activation(T *, const T *, const T *, int, int, int, __nv_bool, cudaStream_t) [with T=__half]" at line 706

/data/v-yulinli/workspace/anaconda3/envs/deepspeed/lib/python3.10/site-packages/deepspeed/ops/csrc/transformer/inference/csrc/gelu.cu(652): error: no operator "+" matches these operands
operand types are: __half + __half
T pre_gate = activation_buffer_2[v] + bias_buffer_2[v];
^
detected during:
instantiation of "void fused_gate_activation<T,useGelu>(T *, const T *, const T *, int, int, int) [with T=__half, useGelu=true]" at line 695
instantiation of "void launch_gated_activation(T *, const T *, const T *, int, int, int, __nv_bool, cudaStream_t) [with T=__half]" at line 706

/data/v-yulinli/workspace/anaconda3/envs/deepspeed/lib/python3.10/site-packages/deepspeed/ops/csrc/transformer/inference/csrc/gelu.cu(656): error: no operator "*" matches these operands
operand types are: __half * __half
activation_buffer_1[v] = hidden_state * gate;
^
detected during:
instantiation of "void fused_gate_activation<T,useGelu>(T *, const T *, const T *, int, int, int) [with T=__half, useGelu=true]" at line 695
instantiation of "void launch_gated_activation(T *, const T *, const T *, int, int, int, __nv_bool, cudaStream_t) [with T=__half]" at line 706

/data/v-yulinli/workspace/anaconda3/envs/deepspeed/lib/python3.10/site-packages/deepspeed/ops/csrc/transformer/inference/csrc/gelu.cu(651): error: no operator "+" matches these operands
operand types are: __half + __half
T hidden_state = activation_buffer_1[v] + bias_buffer_1[v];
^
detected during:
instantiation of "void fused_gate_activation<T,useGelu>(T *, const T *, const T *, int, int, int) [with T=__half, useGelu=false]" at line 698
instantiation of "void launch_gated_activation(T *, const T *, const T *, int, int, int, __nv_bool, cudaStream_t) [with T=__half]" at line 706

/data/v-yulinli/workspace/anaconda3/envs/deepspeed/lib/python3.10/site-packages/deepspeed/ops/csrc/transformer/inference/csrc/gelu.cu(652): error: no operator "+" matches these operands
operand types are: __half + __half
T pre_gate = activation_buffer_2[v] + bias_buffer_2[v];
^
detected during:
instantiation of "void fused_gate_activation<T,useGelu>(T *, const T *, const T *, int, int, int) [with T=__half, useGelu=false]" at line 698
instantiation of "void launch_gated_activation(T *, const T *, const T *, int, int, int, __nv_bool, cudaStream_t) [with T=__half]" at line 706

/data/v-yulinli/workspace/anaconda3/envs/deepspeed/lib/python3.10/site-packages/deepspeed/ops/csrc/transformer/inference/csrc/gelu.cu(656): error: no operator "*" matches these operands
operand types are: __half * __half
activation_buffer_1[v] = hidden_state * gate;
^
detected during:
instantiation of "void fused_gate_activation<T,useGelu>(T *, const T *, const T *, int, int, int) [with T=__half, useGelu=false]" at line 698
instantiation of "void launch_gated_activation(T *, const T *, const T *, int, int, int, __nv_bool, cudaStream_t) [with T=__half]" at line 706

8 errors detected in the compilation of "/data/v-yulinli/workspace/anaconda3/envs/deepspeed/lib/python3.10/site-packages/deepspeed/ops/csrc/transformer/inference/csrc/gelu.cu".
ninja: build stopped: subcommand failed.
Traceback (most recent call last):
File "/data/v-yulinli/workspace/anaconda3/envs/deepspeed/lib/python3.10/site-packages/torch/utils/cpp_extension.py", line 1893, in _run_ninja_build
subprocess.run(
File "/data/v-yulinli/workspace/anaconda3/envs/deepspeed/lib/python3.10/subprocess.py", line 526, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/data/v-yulinli/workspace/deepspeed_chat/DeepSpeedExamples/applications/DeepSpeed-Chat/training/step3_rlhf_finetuning/main.py", line 631, in
main()
File "/data/v-yulinli/workspace/deepspeed_chat/DeepSpeedExamples/applications/DeepSpeed-Chat/training/step3_rlhf_finetuning/main.py", line 447, in main
rlhf_engine = DeepSpeedRLHFEngine(
File "/data/v-yulinli/workspace/deepspeed_chat/DeepSpeedExamples/applications/DeepSpeed-Chat/training/step3_rlhf_finetuning/rlhf_engine.py", line 48, in init
self.actor = self._init_actor(
File "/data/v-yulinli/workspace/deepspeed_chat/DeepSpeedExamples/applications/DeepSpeed-Chat/training/step3_rlhf_finetuning/rlhf_engine.py", line 125, in init_actor
actor_engine, * = deepspeed.initialize(model=actor_model,
File "/data/v-yulinli/workspace/anaconda3/envs/deepspeed/lib/python3.10/site-packages/deepspeed/init.py", line 159, in initialize
engine = DeepSpeedHybridEngine(args=args,
File "/data/v-yulinli/workspace/anaconda3/envs/deepspeed/lib/python3.10/site-packages/deepspeed/runtime/hybrid_engine.py", line 52, in init
self.create_inference_module()
File "/data/v-yulinli/workspace/anaconda3/envs/deepspeed/lib/python3.10/site-packages/deepspeed/runtime/hybrid_engine.py", line 355, in create_inference_module
self.create_inference_containers(self.module)
File "/data/v-yulinli/workspace/anaconda3/envs/deepspeed/lib/python3.10/site-packages/deepspeed/runtime/hybrid_engine.py", line 304, in create_inference_containers
self.create_inference_containers(child, layer_id=layer_id)
File "/data/v-yulinli/workspace/anaconda3/envs/deepspeed/lib/python3.10/site-packages/deepspeed/runtime/hybrid_engine.py", line 304, in create_inference_containers
self.create_inference_containers(child, layer_id=layer_id)
File "/data/v-yulinli/workspace/anaconda3/envs/deepspeed/lib/python3.10/site-packages/deepspeed/runtime/hybrid_engine.py", line 304, in create_inference_containers
self.create_inference_containers(child, layer_id=layer_id)
File "/data/v-yulinli/workspace/anaconda3/envs/deepspeed/lib/python3.10/site-packages/deepspeed/runtime/hybrid_engine.py", line 284, in create_inference_containers
self._inference_containers.append(self.inference_policies[child.class][0](
File "/data/v-yulinli/workspace/anaconda3/envs/deepspeed/lib/python3.10/site-packages/deepspeed/runtime/hybrid_engine.py", line 118, in new_inference_container
_container.create_module()
File "/data/v-yulinli/workspace/anaconda3/envs/deepspeed/lib/python3.10/site-packages/deepspeed/module_inject/containers/opt.py", line 28, in create_module
self.module = DeepSpeedOPTInference(_config, mp_group=self.mp_group)
File "/data/v-yulinli/workspace/anaconda3/envs/deepspeed/lib/python3.10/site-packages/deepspeed/model_implementations/transformers/ds_opt.py", line 20, in init
super().init(config, mp_group, quantize_scales, quantize_groups, merge_count, mlp_extra_grouping)
File "/data/v-yulinli/workspace/anaconda3/envs/deepspeed/lib/python3.10/site-packages/deepspeed/model_implementations/transformers/ds_transformer.py", line 58, in init
inference_module = builder.load()
File "/data/v-yulinli/workspace/anaconda3/envs/deepspeed/lib/python3.10/site-packages/deepspeed/ops/op_builder/builder.py", line 446, in load
return self.jit_load(verbose)
File "/data/v-yulinli/workspace/anaconda3/envs/deepspeed/lib/python3.10/site-packages/deepspeed/ops/op_builder/builder.py", line 489, in jit_load
op_module = load(name=self.name,
File "/data/v-yulinli/workspace/anaconda3/envs/deepspeed/lib/python3.10/site-packages/torch/utils/cpp_extension.py", line 1284, in load
return _jit_compile(
File "/data/v-yulinli/workspace/anaconda3/envs/deepspeed/lib/python3.10/site-packages/torch/utils/cpp_extension.py", line 1509, in _jit_compile
_write_ninja_file_and_build_library(
File "/data/v-yulinli/workspace/anaconda3/envs/deepspeed/lib/python3.10/site-packages/torch/utils/cpp_extension.py", line 1624, in _write_ninja_file_and_build_library
_run_ninja_build(
File "/data/v-yulinli/workspace/anaconda3/envs/deepspeed/lib/python3.10/site-packages/torch/utils/cpp_extension.py", line 1909, in _run_ninja_build
raise RuntimeError(message) from e
RuntimeError: Error building extension 'transformer_inference'

hi, how to solve this problem?

Start with the reported reduction_utils.h errors and the layer_norm.cu and rms_norm.cu compilation commands under deepspeed/ops/csrc/transformer/inference/csrc. Reproduce the extension build in the shown Python 3.10, PyTorch, and CUDA environment, then determine the expected compatibility fix. Done means transformer_inference builds without the reported half and half2 operator errors.