GPU GEOS : Mismatched template definition usage in coreComponents/physicsSolvers
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 287
- Forks
- 109
- Avg merge
- 4d 41m
- Merged PRs (30d)
- 5
Description
Describe the bug
Building GEOS (latest "develop" and TPL "master") with CUDA, runs into conflicting definitions for certain templates.
To Reproduce
Steps to reproduce the behavior:
- Build TPL and then GEOS with GCC 13.2.0 and CUDA 11.8 or 12.5, or 12.6.
- Click on '....'
- Scroll down to '....'
- See error
Try to provide minimal test cases where possible to help isolate the problem.
Expected behavior
A clear and concise description of what you expected to happen.
Screenshots
If applicable, add screenshots to help explain your problem.
Platform (please complete the following information):
- Machine AMD node CPU : AMD EPYC 7V12 and GPU: A100 A100-SXM4-80GB
- Compiler: Host: gcc 13.2.0, GPU : CUDA 11.8 or 12.5, or 12.6
- GEOS Version : v1.1.0 commit 93f0252a19f240dc3b7375cd93bbe036ac318063
Additional context
Output from make all
[ 86%] Building CUDA object coreComponents/physicsSolvers/CMakeFiles/physicsSolvers.dir/wavePropagation/sem/acoustic/secondOrderEqn/anisotropic/AcousticVTIWaveEquationSEM.cpp.o
cd /dev/shm/mtml/src/GEOS/GEOS/build-GPU-Hypre-GCC-CUDA_12.6-ompi_hpcx-OMP-relwithdebinfo/coreComponents/physicsSolvers && /vend/nvidia/cuda/v12.6/bin/nvcc -forward-unknown-to-host-compiler -ccbin=/chv/az_ussc_p/x86_64-rhel3/util/hpcx/hpcx-v2.20-gcc-mlnx_ofed-redhat8-cuda12-x86_64/ompi/bin/mpic++ -DCAMP_HAVE_CUDA -DDIY_NO_THREADS -DFMT_HEADER_ONLY=1 -DH5_BUILT_AS_DYNAMIC_LIB -DMPICH_SKIP_MPICXX -DMPI_NO_CPPBIND -DOMPI_SKIP_MPICXX -D_FILE_OFFSET_BITS=64 -D_GNU_SOURCE -D_LARGEFILE64_SOURCE -D_LARGEFILE_SOURCE -D_MPICC_H -D_POSIX_C_SOURCE=200809L -Dkiss_fft_scalar=double -DphysicsSolvers_EXPORTS --options-file CMakeFiles/physicsSolvers.dir/includes_CUDA.rsp -restrict -arch sm_80 --expt-extended-lambda --expt-relaxed-constexpr -Werror cross-execution-space-call,reorder,deprecated-declarations -g -lineinfo -restrict -arch sm_80 --expt-extended-lambda --expt-relaxed-constexpr -Werror cross-execution-space-call,reorder,deprecated-declarations -ftz=true -O3 -DNDEBUG -Xcompiler -DNDEBUG -Xcompiler -Ofast -ftz=true -std=c++17 "--generate-code=arch=compute_80,code=[compute_80,sm_80]" -Xcompiler=-fPIC -Xcompiler=-fopenmp -Xcompiler=-pthread -Xcompiler -pthread -MD -MT coreComponents/physicsSolvers/CMakeFiles/physicsSolvers.dir/wavePropagation/sem/acoustic/secondOrderEqn/anisotropic/AcousticVTIWaveEquationSEM.cpp.o -MF CMakeFiles/physicsSolvers.dir/wavePropagation/sem/acoustic/secondOrderEqn/anisotropic/AcousticVTIWaveEquationSEM.cpp.o.d -x cu -rdc=true -c /dev/shm/mtml/src/GEOS/GEOS/src/coreComponents/physicsSolvers/wavePropagation/sem/acoustic/secondOrderEqn/anisotropic/AcousticVTIWaveEquationSEM.cpp -o CMakeFiles/physicsSolvers.dir/wavePropagation/sem/acoustic/secondOrderEqn/anisotropic/AcousticVTIWaveEquationSEM.cpp.o
[ 86%] Building CUDA object coreComponents/physicsSolvers/CMakeFiles/physicsSolvers.dir/wavePropagation/sem/acoustoelastic/secondOrderEqn/isotropic/AcousticElasticWaveEquationSEM.cpp.o
cd /dev/shm/mtml/src/GEOS/GEOS/build-GPU-Hypre-GCC-CUDA_12.6-ompi_hpcx-OMP-relwithdebinfo/coreComponents/physicsSolvers && /vend/nvidia/cuda/v12.6/bin/nvcc -forward-unknown-to-host-compiler -ccbin=/chv/az_ussc_p/x86_64-rhel3/util/hpcx/hpcx-v2.20-gcc-mlnx_ofed-redhat8-cuda12-x86_64/ompi/bin/mpic++ -DCAMP_HAVE_CUDA -DDIY_NO_THREADS -DFMT_HEADER_ONLY=1 -DH5_BUILT_AS_DYNAMIC_LIB -DMPICH_SKIP_MPICXX -DMPI_NO_CPPBIND -DOMPI_SKIP_MPICXX -D_FILE_OFFSET_BITS=64 -D_GNU_SOURCE -D_LARGEFILE64_SOURCE -D_LARGEFILE_SOURCE -D_MPICC_H -D_POSIX_C_SOURCE=200809L -Dkiss_fft_scalar=double -DphysicsSolvers_EXPORTS --options-file CMakeFiles/physicsSolvers.dir/includes_CUDA.rsp -restrict -arch sm_80 --expt-extended-lambda --expt-relaxed-constexpr -Werror cross-execution-space-call,reorder,deprecated-declarations -g -lineinfo -restrict -arch sm_80 --expt-extended-lambda --expt-relaxed-constexpr -Werror cross-execution-space-call,reorder,deprecated-declarations -ftz=true -O3 -DNDEBUG -Xcompiler -DNDEBUG -Xcompiler -Ofast -ftz=true -std=c++17 "--generate-code=arch=compute_80,code=[compute_80,sm_80]" -Xcompiler=-fPIC -Xcompiler=-fopenmp -Xcompiler=-pthread -Xcompiler -pthread -MD -MT coreComponents/physicsSolvers/CMakeFiles/physicsSolvers.dir/wavePropagation/sem/acoustoelastic/secondOrderEqn/isotropic/AcousticElasticWaveEquationSEM.cpp.o -MF CMakeFiles/physicsSolvers.dir/wavePropagation/sem/acoustoelastic/secondOrderEqn/isotropic/AcousticElasticWaveEquationSEM.cpp.o.d -x cu -rdc=true -c /dev/shm/mtml/src/GEOS/GEOS/src/coreComponents/physicsSolvers/wavePropagation/sem/acoustoelastic/secondOrderEqn/isotropic/AcousticElasticWaveEquationSEM.cpp -o CMakeFiles/physicsSolvers.dir/wavePropagation/sem/acoustoelastic/secondOrderEqn/isotropic/AcousticElasticWaveEquationSEM.cpp.o
[ 86%] Linking CUDA device code CMakeFiles/physicsSolvers.dir/cmake_device_link.o
cd /dev/shm/mtml/src/GEOS/GEOS/build-GPU-Hypre-GCC-CUDA_12.6-ompi_hpcx-OMP-relwithdebinfo/coreComponents/physicsSolvers && /data/saet/mtml/software/x86_64/cmake-3.28.3-linux-x86_64/bin/cmake -E cmake_link_script CMakeFiles/physicsSolvers.dir/dlink.txt --verbose=1
/vend/nvidia/cuda/v12.6/bin/nvcc -forward-unknown-to-host-compiler -ccbin=/chv/az_ussc_p/x86_64-rhel3/util/hpcx/hpcx-v2.20-gcc-mlnx_ofed-redhat8-cuda12-x86_64/ompi/bin/mpic++ -restrict -arch sm_80 --expt-extended-lambda --expt-relaxed-constexpr -Werror cross-execution-space-call,reorder,deprecated-declarations -g -lineinfo -restrict -arch sm_80 --expt-extended-lambda --expt-relaxed-constexpr -Werror cross-execution-space-call,reorder,deprecated-declarations -ftz=true -O3 -DNDEBUG -Xcompiler -DNDEBUG -Xcompiler -Ofast -ftz=true "--generate-code=arch=compute_80,code=[compute_80,sm_80]" -Xcompiler=-fopenmp -Xcompiler=-L/vend/nvidia/cuda/v12.6/lib64 -Xlinker=-rpath -Xlinker=/chv/az_ussc_p/x86_64-rhel3/util/hpcx/hpcx-v2.20-gcc-mlnx_ofed-redhat8-cuda12-x86_64/ompi/lib -Xlinker=--enable-new-dtags -Xcompiler=-pthread -Xcompiler=-fPIC -Wno-deprecated-gpu-targets -shared -dlink --options-file CMakeFiles/physicsSolvers.dir/deviceObjects1.rsp --options-file CMakeFiles/physicsSolvers.dir/deviceObjects2.rsp -o CMakeFiles/physicsSolvers.dir/cmake_device_link.o --options-file CMakeFiles/physicsSolvers.dir/deviceLinkLibs.rsp
nvlink error : Size doesn't match for '_ZN4geos13finiteElement17FiniteElementBaseC1ERKS1_$824' in 'CMakeFiles/physicsSolvers.dir/__/__/generatedSrc/coreComponents/physicsSolvers/multiphysics/poromechanicsKernels/PoromechanicsKernels_CellElementSubRegion_PorousSolid-DelftEgg-_H1_Hexahedron_Lagrange1_GaussLegendre2.cpp.o', first specified in 'CMakeFiles/physicsSolvers.dir/__/__/generatedSrc/coreComponents/physicsSolvers/multiphysics/poromechanicsKernels/PoromechanicsKernels_CellElementSubRegion_PorousSolid-DruckerPragerExtended-_H1_Hexahedron_Lagrange1_GaussLegendre2.cpp.o' (target: sm_80)
nvlink fatal : merge_elf failed (target: sm_80)
make[2]: *** [coreComponents/physicsSolvers/CMakeFiles/physicsSolvers.dir/build.make:13103: coreComponents/physicsSolvers/CMakeFiles/physicsSolvers.dir/cmake_device_link.o] Error 1
make[2]: Target 'coreComponents/physicsSolvers/CMakeFiles/physicsSolvers.dir/build' not remade because of errors.
make[2]: Leaving directory '/dev/shm/mtml/src/GEOS/GEOS/build-GPU-Hypre-GCC-CUDA_12.6-ompi_hpcx-OMP-relwithdebinfo'
make[1]: *** [CMakeFiles/Makefile2:8964: coreComponents/physicsSolvers/CMakeFiles/physicsSolvers.dir/all] Error 2
make[1]: Target 'all' not remade because of errors.
make[1]: Leaving directory '/dev/shm/mtml/src/GEOS/GEOS/build-GPU-Hypre-GCC-CUDA_12.6-ompi_hpcx-OMP-relwithdebinfo'
make: *** [Makefile:146: all] Error 2
[Make-build-2024-12-17-103544.log](https://github.com/user-attachments/files/18171272/Make-build-2024-12-17-103544.log)
[PoromechanicsKernels_CellElementSubRegion_PorousSolid-DruckerPragerExtended-_H1_Hexahedron_Lagrange1_GaussLegendre2.cpp.txt](https://github.com/user-attachments/files/18171302/PoromechanicsKernels_CellElementSubRegion_PorousSolid-DruckerPragerExtended-_H1_Hexahedron_Lagrange1_GaussLegendre2.cpp.txt)
[ThermoPoromechanicsKernels_CellElementSubRegion_PorousSolid-DelftEgg-_H1_Hexahedron_Lagrange1_GaussLegendre2.cpp.txt](https://github.com/user-attachments/files/18171303/ThermoPoromechanicsKernels_CellElementSubRegion_PorousSolid-DelftEgg-_H1_Hexahedron_Lagrange1_GaussLegendre2.cpp.txt)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the CUDA build in coreComponents/physicsSolvers using the compiler and CUDA versions listed in the issue, then inspect the generated PoromechanicsKernels and ThermoPoromechanicsKernels source attachments. Compare the template definitions implicated by the nvlink error and confirm the fix by completing the physicsSolvers device-link step successfully.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- hpc
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100