GEOS-DEV / GEOS-DEV/GEOS

GPU GEOS : Mismatched template definition usage in coreComponents/physicsSolvers

Open
#3,496 13 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

type: bug type: new
Dominant language
C++
Stars
287
Forks
109
Avg merge
4d 41m
Merged PRs (30d)
5

Description

Describe the bug
Building GEOS (latest "develop" and TPL "master") with CUDA, runs into conflicting definitions for certain templates.

To Reproduce
Steps to reproduce the behavior:

  1. Build TPL and then GEOS with GCC 13.2.0 and CUDA 11.8 or 12.5, or 12.6.
  2. Click on '....'
  3. Scroll down to '....'
  4. See error
    Try to provide minimal test cases where possible to help isolate the problem.

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Platform (please complete the following information):

  • Machine AMD node CPU : AMD EPYC 7V12 and GPU: A100 A100-SXM4-80GB
  • Compiler: Host: gcc 13.2.0, GPU : CUDA 11.8 or 12.5, or 12.6
  • GEOS Version : v1.1.0 commit 93f0252a19f240dc3b7375cd93bbe036ac318063

Additional context
Output from make all

[ 86%] Building CUDA object coreComponents/physicsSolvers/CMakeFiles/physicsSolvers.dir/wavePropagation/sem/acoustic/secondOrderEqn/anisotropic/AcousticVTIWaveEquationSEM.cpp.o
cd /dev/shm/mtml/src/GEOS/GEOS/build-GPU-Hypre-GCC-CUDA_12.6-ompi_hpcx-OMP-relwithdebinfo/coreComponents/physicsSolvers && /vend/nvidia/cuda/v12.6/bin/nvcc -forward-unknown-to-host-compiler -ccbin=/chv/az_ussc_p/x86_64-rhel3/util/hpcx/hpcx-v2.20-gcc-mlnx_ofed-redhat8-cuda12-x86_64/ompi/bin/mpic++ -DCAMP_HAVE_CUDA -DDIY_NO_THREADS -DFMT_HEADER_ONLY=1 -DH5_BUILT_AS_DYNAMIC_LIB -DMPICH_SKIP_MPICXX -DMPI_NO_CPPBIND -DOMPI_SKIP_MPICXX -D_FILE_OFFSET_BITS=64 -D_GNU_SOURCE -D_LARGEFILE64_SOURCE -D_LARGEFILE_SOURCE -D_MPICC_H -D_POSIX_C_SOURCE=200809L -Dkiss_fft_scalar=double -DphysicsSolvers_EXPORTS --options-file CMakeFiles/physicsSolvers.dir/includes_CUDA.rsp -restrict -arch sm_80 --expt-extended-lambda --expt-relaxed-constexpr -Werror cross-execution-space-call,reorder,deprecated-declarations   -g -lineinfo    -restrict -arch sm_80 --expt-extended-lambda --expt-relaxed-constexpr -Werror cross-execution-space-call,reorder,deprecated-declarations  -ftz=true -O3 -DNDEBUG -Xcompiler -DNDEBUG -Xcompiler -Ofast -ftz=true   -std=c++17 "--generate-code=arch=compute_80,code=[compute_80,sm_80]" -Xcompiler=-fPIC -Xcompiler=-fopenmp -Xcompiler=-pthread -Xcompiler -pthread -MD -MT coreComponents/physicsSolvers/CMakeFiles/physicsSolvers.dir/wavePropagation/sem/acoustic/secondOrderEqn/anisotropic/AcousticVTIWaveEquationSEM.cpp.o -MF CMakeFiles/physicsSolvers.dir/wavePropagation/sem/acoustic/secondOrderEqn/anisotropic/AcousticVTIWaveEquationSEM.cpp.o.d -x cu -rdc=true -c /dev/shm/mtml/src/GEOS/GEOS/src/coreComponents/physicsSolvers/wavePropagation/sem/acoustic/secondOrderEqn/anisotropic/AcousticVTIWaveEquationSEM.cpp -o CMakeFiles/physicsSolvers.dir/wavePropagation/sem/acoustic/secondOrderEqn/anisotropic/AcousticVTIWaveEquationSEM.cpp.o
[ 86%] Building CUDA object coreComponents/physicsSolvers/CMakeFiles/physicsSolvers.dir/wavePropagation/sem/acoustoelastic/secondOrderEqn/isotropic/AcousticElasticWaveEquationSEM.cpp.o
cd /dev/shm/mtml/src/GEOS/GEOS/build-GPU-Hypre-GCC-CUDA_12.6-ompi_hpcx-OMP-relwithdebinfo/coreComponents/physicsSolvers && /vend/nvidia/cuda/v12.6/bin/nvcc -forward-unknown-to-host-compiler -ccbin=/chv/az_ussc_p/x86_64-rhel3/util/hpcx/hpcx-v2.20-gcc-mlnx_ofed-redhat8-cuda12-x86_64/ompi/bin/mpic++ -DCAMP_HAVE_CUDA -DDIY_NO_THREADS -DFMT_HEADER_ONLY=1 -DH5_BUILT_AS_DYNAMIC_LIB -DMPICH_SKIP_MPICXX -DMPI_NO_CPPBIND -DOMPI_SKIP_MPICXX -D_FILE_OFFSET_BITS=64 -D_GNU_SOURCE -D_LARGEFILE64_SOURCE -D_LARGEFILE_SOURCE -D_MPICC_H -D_POSIX_C_SOURCE=200809L -Dkiss_fft_scalar=double -DphysicsSolvers_EXPORTS --options-file CMakeFiles/physicsSolvers.dir/includes_CUDA.rsp -restrict -arch sm_80 --expt-extended-lambda --expt-relaxed-constexpr -Werror cross-execution-space-call,reorder,deprecated-declarations   -g -lineinfo    -restrict -arch sm_80 --expt-extended-lambda --expt-relaxed-constexpr -Werror cross-execution-space-call,reorder,deprecated-declarations  -ftz=true -O3 -DNDEBUG -Xcompiler -DNDEBUG -Xcompiler -Ofast -ftz=true   -std=c++17 "--generate-code=arch=compute_80,code=[compute_80,sm_80]" -Xcompiler=-fPIC -Xcompiler=-fopenmp -Xcompiler=-pthread -Xcompiler -pthread -MD -MT coreComponents/physicsSolvers/CMakeFiles/physicsSolvers.dir/wavePropagation/sem/acoustoelastic/secondOrderEqn/isotropic/AcousticElasticWaveEquationSEM.cpp.o -MF CMakeFiles/physicsSolvers.dir/wavePropagation/sem/acoustoelastic/secondOrderEqn/isotropic/AcousticElasticWaveEquationSEM.cpp.o.d -x cu -rdc=true -c /dev/shm/mtml/src/GEOS/GEOS/src/coreComponents/physicsSolvers/wavePropagation/sem/acoustoelastic/secondOrderEqn/isotropic/AcousticElasticWaveEquationSEM.cpp -o CMakeFiles/physicsSolvers.dir/wavePropagation/sem/acoustoelastic/secondOrderEqn/isotropic/AcousticElasticWaveEquationSEM.cpp.o

[ 86%] Linking CUDA device code CMakeFiles/physicsSolvers.dir/cmake_device_link.o
cd /dev/shm/mtml/src/GEOS/GEOS/build-GPU-Hypre-GCC-CUDA_12.6-ompi_hpcx-OMP-relwithdebinfo/coreComponents/physicsSolvers && /data/saet/mtml/software/x86_64/cmake-3.28.3-linux-x86_64/bin/cmake -E cmake_link_script CMakeFiles/physicsSolvers.dir/dlink.txt --verbose=1
/vend/nvidia/cuda/v12.6/bin/nvcc -forward-unknown-to-host-compiler -ccbin=/chv/az_ussc_p/x86_64-rhel3/util/hpcx/hpcx-v2.20-gcc-mlnx_ofed-redhat8-cuda12-x86_64/ompi/bin/mpic++   -restrict -arch sm_80 --expt-extended-lambda --expt-relaxed-constexpr -Werror cross-execution-space-call,reorder,deprecated-declarations   -g -lineinfo    -restrict -arch sm_80 --expt-extended-lambda --expt-relaxed-constexpr -Werror cross-execution-space-call,reorder,deprecated-declarations  -ftz=true -O3 -DNDEBUG -Xcompiler -DNDEBUG -Xcompiler -Ofast -ftz=true   "--generate-code=arch=compute_80,code=[compute_80,sm_80]" -Xcompiler=-fopenmp -Xcompiler=-L/vend/nvidia/cuda/v12.6/lib64 -Xlinker=-rpath -Xlinker=/chv/az_ussc_p/x86_64-rhel3/util/hpcx/hpcx-v2.20-gcc-mlnx_ofed-redhat8-cuda12-x86_64/ompi/lib -Xlinker=--enable-new-dtags -Xcompiler=-pthread -Xcompiler=-fPIC -Wno-deprecated-gpu-targets -shared -dlink --options-file CMakeFiles/physicsSolvers.dir/deviceObjects1.rsp --options-file CMakeFiles/physicsSolvers.dir/deviceObjects2.rsp -o CMakeFiles/physicsSolvers.dir/cmake_device_link.o --options-file CMakeFiles/physicsSolvers.dir/deviceLinkLibs.rsp
nvlink error   : Size doesn't match for '_ZN4geos13finiteElement17FiniteElementBaseC1ERKS1_$824' in 'CMakeFiles/physicsSolvers.dir/__/__/generatedSrc/coreComponents/physicsSolvers/multiphysics/poromechanicsKernels/PoromechanicsKernels_CellElementSubRegion_PorousSolid-DelftEgg-_H1_Hexahedron_Lagrange1_GaussLegendre2.cpp.o', first specified in 'CMakeFiles/physicsSolvers.dir/__/__/generatedSrc/coreComponents/physicsSolvers/multiphysics/poromechanicsKernels/PoromechanicsKernels_CellElementSubRegion_PorousSolid-DruckerPragerExtended-_H1_Hexahedron_Lagrange1_GaussLegendre2.cpp.o' (target: sm_80)
nvlink fatal   : merge_elf failed (target: sm_80)
make[2]: *** [coreComponents/physicsSolvers/CMakeFiles/physicsSolvers.dir/build.make:13103: coreComponents/physicsSolvers/CMakeFiles/physicsSolvers.dir/cmake_device_link.o] Error 1
make[2]: Target 'coreComponents/physicsSolvers/CMakeFiles/physicsSolvers.dir/build' not remade because of errors.
make[2]: Leaving directory '/dev/shm/mtml/src/GEOS/GEOS/build-GPU-Hypre-GCC-CUDA_12.6-ompi_hpcx-OMP-relwithdebinfo'
make[1]: *** [CMakeFiles/Makefile2:8964: coreComponents/physicsSolvers/CMakeFiles/physicsSolvers.dir/all] Error 2
make[1]: Target 'all' not remade because of errors.
make[1]: Leaving directory '/dev/shm/mtml/src/GEOS/GEOS/build-GPU-Hypre-GCC-CUDA_12.6-ompi_hpcx-OMP-relwithdebinfo'
make: *** [Makefile:146: all] Error 2
[Make-build-2024-12-17-103544.log](https://github.com/user-attachments/files/18171272/Make-build-2024-12-17-103544.log)
[PoromechanicsKernels_CellElementSubRegion_PorousSolid-DruckerPragerExtended-_H1_Hexahedron_Lagrange1_GaussLegendre2.cpp.txt](https://github.com/user-attachments/files/18171302/PoromechanicsKernels_CellElementSubRegion_PorousSolid-DruckerPragerExtended-_H1_Hexahedron_Lagrange1_GaussLegendre2.cpp.txt)
[ThermoPoromechanicsKernels_CellElementSubRegion_PorousSolid-DelftEgg-_H1_Hexahedron_Lagrange1_GaussLegendre2.cpp.txt](https://github.com/user-attachments/files/18171303/ThermoPoromechanicsKernels_CellElementSubRegion_PorousSolid-DelftEgg-_H1_Hexahedron_Lagrange1_GaussLegendre2.cpp.txt)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the CUDA build in coreComponents/physicsSolvers using the compiler and CUDA versions listed in the issue, then inspect the generated PoromechanicsKernels and ThermoPoromechanicsKernels source attachments. Compare the template definitions implicated by the nvlink error and confirm the fix by completing the physicsSolvers device-link step successfully.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
hpc
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.