Combined Intel+NVIDIA+AMD Build Crashes Compiler (2024.03.13.)
- Dominant language
- LLVM
- Stars
- 1.5k
- Forks
- 854
- Avg merge
- 3d 17h
- Merged PRs (30d)
- 137
Description
### Describe the bug
Recently, with help from @fwyzard and Aurora (I don't know her GitHub username unfortunately... :frowning:), we did some experiments with building hybrid Intel+NVIDIA+AMD binaries using the oneAPI compiler. The winning incantation being of the following kind:
```
icpx -fsycl \
-fsycl-targets=spir64,spir64_x86_64,nvidia_gpu_sm_86,amd_gpu_gfx803 \
-Xclang -opaque-pointers \
-Wno-unknown-cuda-version
```
This works well for "simple enough" source code. :wink: But similar to #8065, once I try to build our "most complicated" project, things break. :frowning: However, with the current latest version of oneAPI (2024.0.1), the failure is different than what I described in #8065. During the build I get:
```
...
[ 88%] Building SYCL object device/sycl/CMakeFiles/traccc_sycl.dir/src/fitting/fitting_algorithm.sycl.o
cd /data/ssd-1tb/projects/traccc/build/device/sycl && /home/krasznaa/software/intel/oneapi-2024.0.1/compiler/2024.0/bin/icpx -x c++ -DALGEBRA_PLUGINS_INCLUDE_ARRAY -DCOVFIE_QUIET -DDETRAY_ALGEBRA_ARRAY -DDETRAY_ALGEBRA_EIGEN -DDETRAY_ALGEBRA_VC -DDETRAY_CUSTOM_SCALARTYPE=float -DEIGEN_NO_CUDA -DEIGEN_NO_HIP -DTHRUST_DEVICE_SYSTEM=THRUST_DEVICE_SYSTEM_CUDA -DTHRUST_HOST_SYSTEM=THRUST_HOST_SYSTEM_CPP -DTRACCC_CUSTOM_SCALARTYPE=float -DVECMEM_DEBUG_MSG_LVL=0 -DVECMEM_HAVE_PMR_MEMORY_RESOURCE -DVECMEM_HAVE_SYCL_ATOMIC_REF -DVECMEM_HAVE_SYCL_LOCAL_ACCESSOR -DVECMEM_SOURCE_DIR_LENGTH=37 -DVECMEM_SYCL_PRINTF_FUNCTION=cl::sycl::ext::oneapi::experimental::printf -Dtraccc_sycl_EXPORTS -I/data/ssd-1tb/projects/traccc/traccc/device/sycl/include -I/data/ssd-1tb/projects/traccc/traccc/core/include -I/data/ssd-1tb/projects/traccc/build/_deps/vecmem-build/core/CMakeFiles -I/data/ssd-1tb/projects/traccc/build/_deps/vecmem-src/core/include -I/data/ssd-1tb/projects/traccc/build/_deps/detray-src/core/include -I/data/ssd-1tb/projects/traccc/build/_deps/detray-build/core/CMakeFiles -I/data/ssd-1tb/projects/traccc/build/_deps/thrust-src/thrust/cmake/../.. -I/data/ssd-1tb/projects/traccc/build/_deps/thrust-src/dependencies/libcudacxx/include -I/data/ssd-1tb/projects/traccc/build/_deps/thrust-src/dependencies/cub/cub/cmake/../.. -I/data/ssd-1tb/projects/traccc/traccc/plugins/algebra/array/include -I/data/ssd-1tb/projects/traccc/build/_deps/algebraplugins-src/frontend/array_cmath/include -I/data/ssd-1tb/projects/traccc/build/_deps/algebraplugins-src/common/include -I/data/ssd-1tb/projects/traccc/build/_deps/algebraplugins-src/storage/array/include -I/data/ssd-1tb/projects/traccc/build/_deps/algebraplugins-src/math/cmath/include -I/data/ssd-1tb/projects/traccc/build/_deps/algebraplugins-src/math/common/include -I/data/ssd-1tb/projects/traccc/build/_deps/detray-src/plugins/algebra/array/include -I/data/ssd-1tb/projects/traccc/traccc/plugins/algebra/vecmem/include -I/data/ssd-1tb/projects/traccc/build/_deps/algebraplugins-src/frontend/vecmem_cmath/include -I/data/ssd-1tb/projects/traccc/build/_deps/algebraplugins-src/storage/vecmem/include -I/data/ssd-1tb/projects/traccc/traccc/plugins/algebra/eigen/include -I/data/ssd-1tb/projects/traccc/build/_deps/algebraplugins-src/frontend/eigen_eigen/include -I/data/ssd-1tb/projects/traccc/build/_deps/algebraplugins-src/storage/eigen/include -I/data/ssd-1tb/projects/traccc/build/_deps/algebraplugins-src/math/eigen/include -I/data/ssd-1tb/projects/traccc/build/_deps/detray-src/plugins/algebra/eigen/include -I/data/ssd-1tb/projects/traccc/traccc/plugins/algebra/vc/include -I/data/ssd-1tb/projects/traccc/build/_deps/algebraplugins-src/frontend/vc_vc/include -I/data/ssd-1tb/projects/traccc/build/_deps/algebraplugins-src/storage/vc/include -I/data/ssd-1tb/projects/traccc/build/_deps/algebraplugins-src/math/vc/include -I/data/ssd-1tb/projects/traccc/build/_deps/detray-src/plugins/algebra/vc/include -I/data/ssd-1tb/projects/traccc/build/_deps/algebraplugins-src/frontend/vc_cmath/include -I/data/ssd-1tb/projects/traccc/build/_deps/detray-src/utils/include -I/data/ssd-1tb/projects/traccc/build/_deps/covfie-src/lib/core -I/data/ssd-1tb/projects/traccc/build/_deps/detray-src/io/include -I/data/ssd-1tb/projects/traccc/build/_deps/nlohmann_json-src/include -I/data/ssd-1tb/projects/traccc/traccc/device/common/include -I/data/ssd-1tb/projects/traccc/build/_deps/vecmem-build/sycl/CMakeFiles -I/data/ssd-1tb/projects/traccc/build/_deps/vecmem-src/sycl/include -isystem /data/ssd-1tb/projects/traccc/build/_deps/eigen3-src -isystem /data/ssd-1tb/projects/traccc/build/_deps/vc-src -fsycl -Wno-unknown-cuda-version -fsycl-targets=spir64,spir64_x86_64,nvidia_gpu_sm_86,amd_gpu_gfx803 -Xclang -opaque-pointers -std=c++17 -O3 -Wall -Wextra -Wno-unknown-cuda-version -Wshadow -Wunused-local-typedefs -pedantic -fPIC -MD -MT device/sycl/CMakeFiles/traccc_sycl.dir/src/fitting/fitting_algorithm.sycl.o -MF CMakeFiles/traccc_sycl.dir/src/fitting/fitting_algorithm.sycl.o.d -o CMakeFiles/traccc_sycl.dir/src/fitting/fitting_algorithm.sycl.o -c /data/ssd-1tb/projects/traccc/traccc/device/sycl/src/fitting/fitting_algorithm.sycl
fatal error: error in backend: SmallVector unable to grow. Requested capacity (97325075862528) is larger than maximum value for size type (4294967295)
icpx: error: clang frontend command failed with exit code 70 (use -v to see invocation)
Intel(R) oneAPI DPC++/C++ Compiler 2024.0.2 (2024.0.2.20231213)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /home/krasznaa/software/intel/oneapi-2024.0.1/compiler/2024.0/bin/compiler
Configuration file: /home/krasznaa/software/intel/oneapi-2024.0.1/compiler/2024.0/bin/compiler/../icpx.cfg
icpx: note: diagnostic msg: Error generating preprocessed source(s).
make[3]: *** [device/sycl/CMakeFiles/traccc_sycl.dir/build.make:88: device/sycl/CMakeFiles/traccc_sycl.dir/src/fitting/fitting_algorithm.sycl.o] Error 1
```
### To reproduce
As in the other ticket, reproducing this "from scratch" is a bit of an involved process unfortunately. :frowning: But since the problem is during compilation in this case, I can "simply" give you the preprocessed sources. :smile: You can download that file (it's pretty big) from:
https://cernbox.cern.ch/s/Dc83iva0Dl2czCF
With that file downloaded, you can reproduce this issue like:
```
icpx -x c++-cpp-output -fsycl -Wno-unknown-cuda-version -fsycl-targets=spir64,spir64_x86_64,nvidia_gpu_sm_86,amd_gpu_gfx803 -Xclang -opaque-pointers -std=c++17 -O3 -Wno-unknown-cuda-version -fPIC -c -o fitting_algorithm.sycl.o fitting_algorithm.sycl.i
```
The thing is, it doesn't **always** crash. :thinking: Just now, I was able to execute:
```
[bash][Legolas]:fitting > /home/krasznaa/software/intel/oneapi-2024.0.1/compiler/2024.0/bin/icpx -x c++-cpp-output -fsycl -Wno-unknown-cuda-version -fsycl-targets=spir64,spir64_x86_64,nvidia_gpu_sm_86,amd_gpu_gfx803 -Xclang -opaque-pointers -std=c++17 -O3 -Wno-unknown-cuda-version -fPIC -c -o fitting_algorithm.sycl.o fitting_algorithm.sycl.i
[bash][Legolas]:fitting >
```
While a little earlier the preprocessed file made my compilation crash as expected. :confused:
```
[bash][Legolas]:sycl > /home/krasznaa/software/intel/oneapi-2024.0.1/compiler/2024.0/bin/icpx -x c++-cpp-output -fsycl -Wno-unknown-cuda-version -fsycl-targets=spir64,spir64_x86_64,nvidia_gpu_sm_86,amd_gpu_gfx803 -Xclang -opaque-pointers -std=c++17 -O3 -Wno-unknown-cuda-version -fPIC -c -o CMakeFiles/traccc_sycl.dir/src/fitting/fitting_algorithm.sycl.o CMakeFiles/traccc_sycl.dir/src/fitting/fitting_algorithm.sycl.i 2>&1 | tee error.log
fatal error: error in backend: SmallVector unable to grow. Requested capacity (109556080997376) is larger than maximum value for size type (4294967295)
icpx: error: clang frontend command failed with exit code 70 (use -v to see invocation)
Intel(R) oneAPI DPC++/C++ Compiler 2024.0.2 (2024.0.2.20231213)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /home/krasznaa/software/intel/oneapi-2024.0.1/compiler/2024.0/bin/compiler
Configuration file: /home/krasznaa/software/intel/oneapi-2024.0.1/compiler/2024.0/bin/compiler/../icpx.cfg
icpx: note: diagnostic msg: Error generating preprocessed source(s) - no preprocessable inputs.
[bash][Legolas]:sycl >
```
(I was in a different subdirectory on that try...)
### Environment
- OS: Linux
- Compiler version:
```
Intel(R) oneAPI DPC++/C++ Compiler 2024.0.2 (2024.0.2.20231213)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /home/krasznaa/software/intel/oneapi-2024.0.1/compiler/2024.0/bin/compiler
Configuration file: /home/krasznaa/software/intel/oneapi-2024.0.1/compiler/2024.0/bin/compiler/../icpx.cfg
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/11
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/12
Selected GCC installation: /usr/lib/gcc/x86_64-linux-gnu/12
Candidate multilib: .;@m64
Selected multilib: .;@m64
Found CUDA installation: /home/krasznaa/software/nvidia/cuda-12.4.0/x86_64, version
Found HIP installation: /opt/rocm-6.0.2, version 6.0.32831
```
Don't be fooled by what the compiler reports for HIP though. :thinking: The oneAPI plugin is set up to use ROCm-5.4.6.
```
[bash][Legolas]:build > ldd -r /home/krasznaa/software/intel/oneapi-2024.0.1/compiler/2024.0/lib/libpi_hip.so
linux-vdso.so.1 (0x00007ffff9462000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f8f0d103000)
libamdhip64.so.5 => /opt/rocm/lib/libamdhip64.so.5 (0x00007f8f0b600000)
libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f8f0b200000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f8f0ae00000)
/lib64/ld-linux-x86-64.so.2 (0x00007f8f0d174000)
libhsa-runtime64.so.1 => /opt/rocm/lib/libhsa-runtime64.so.1 (0x00007f8f0aa00000)
libnuma.so.1 => /lib/x86_64-linux-gnu/libnuma.so.1 (0x00007f8f0d0f4000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f8f0d00d000)
libelf.so.1 => /lib/x86_64-linux-gnu/libelf.so.1 (0x00007f8f0cfed000)
libdrm.so.2 => /opt/amdgpu/lib/x86_64-linux-gnu/libdrm.so.2 (0x00007f8f0cfd3000)
libdrm_amdgpu.so.1 => /opt/amdgpu/lib/x86_64-linux-gnu/libdrm_amdgpu.so.1 (0x00007f8f0cfc5000)
libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f8f0cfa9000)
[bash][Legolas]:build > ls -l /opt/rocm
lrwxrwxrwx 1 root root 22 Sep 5 2023 /opt/rocm -> /etc/alternatives/rocm
[bash][Legolas]:build > ls -l /etc/alternatives/rocm
lrwxrwxrwx 1 root root 15 Sep 30 14:11 /etc/alternatives/rocm -> /opt/rocm-5.4.6
[bash][Legolas]:build >
```
I just happen to do a native HIP build with version 6.0.2 at the same time...
### Additional context
The build for "just" an Intel+NVIDIA backend does work. Only once I try to build for all 3 in parallel, that I see this issue pop up. :thinking:
Contributor guide
Assessment
This issue has not been assessed yet.