[Build] Build Error with MVAPICH2-GDR

Describe the issue

A clear and concise description of what the issue is.

I am trying to build AMGX with MVAPICH2-GDR on the TACC Supercomputer Lonestar6 as the MPI. Building with Intel MPI works fine.

Environment information:

• OS: Rocky Linux release 8.6 (Green Obsidian)
• Compiler version: Intel 19.1.0
• CMake version: 3.24.2
• CUDA used for AMGX compilation: CUDA 12.0
• MPI version (if applicable): MVAPICH2-GDR v2.3.7
• AMGX version or commit hash v 2.4.0 f280993e5df32701fb28c2636f49eea4bc5753a4
• Any related environment variables information

Configuration information

Provide your cmake command line that was used for configuration and it's full output.

cmake -DCMAKE_C_COMPILER=mpicc -DCMAKE_CXX_COMPILER=mpic++ -DCMAKE_CUDA_HOST_COMPILER=mpic++ -DCUDA_ARCH=80 ..

Output

-- The C compiler identification is Intel 19.1.0.20200306
-- The CXX compiler identification is Intel 19.1.0.20200306
-- The CUDA compiler identification is NVIDIA 12.0.140
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /opt/apps/intel19/mvapich2-gdr/2.3.7/bin/mpicc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /opt/apps/intel19/mvapich2-gdr/2.3.7/bin/mpic++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Detecting CUDA compiler ABI info
-- Detecting CUDA compiler ABI info - done
-- Check for working CUDA compiler: /opt/apps/cuda/12.0/bin/nvcc - skipped
-- Detecting CUDA compile features
-- Detecting CUDA compile features - done
-- Found MPI_C: /opt/apps/intel19/mvapich2-gdr/2.3.7/bin/mpicc (found version "3.1") 
-- Found MPI_CXX: /opt/apps/intel19/mvapich2-gdr/2.3.7/bin/mpic++ (found version "3.1") 
-- Found MPI: TRUE (found version "3.1")
-- Found OpenMP_C: -qopenmp (found version "5.0") 
-- Found OpenMP_CXX: -qopenmp (found version "5.0") 
-- Found OpenMP: TRUE (found version "5.0")
-- Found CUDAToolkit: /opt/apps/cuda/12.0/include (found suitable version "12.0.140", minimum required is "10.0")
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE
-- Found OpenMP_C: -qopenmp (found version "5.0") 
-- Found OpenMP_CXX: -qopenmp (found version "5.0")
This is a MPI build:TRUE
-- Found libcudacxx: /work/08435/srvenkat/frontera/AMGX/thrust/dependencies/libcudacxx/lib/cmake/libcudacxx/libcudacxx-config.cmake (found suitable version "1.8.1.0", minimum required is "1.8.0")
-- Found Thrust: /work/08435/srvenkat/frontera/AMGX/thrust/thrust/cmake/thrust-config.cmake (found version "2.1.0.0")
-- Found CUB: /work/08435/srvenkat/frontera/AMGX/thrust/dependencies/cub/cub/cmake/cub-config.cmake (found suitable version "2.1.0.0", minimum required is "2.1.0.0")
-- Configuring done
-- Generating done
-- Build files have been written to: /work/08435/srvenkat/frontera/AMGX/build2

Compilation information

Provide your make command

make all VERBOSE=1

Output

/opt/apps/cmake/3.24.2/bin/cmake -S/work/08435/srvenkat/frontera/AMGX -B/work/08435/srvenkat/frontera/AMGX/build2 --check-build-system CMakeFiles/Makefile.cmake 0
/opt/apps/cmake/3.24.2/bin/cmake -E cmake_progress_start /work/08435/srvenkat/frontera/AMGX/build2/CMakeFiles /work/08435/srvenkat/frontera/AMGX/build2//CMakeFiles/progress.marks
make  -f CMakeFiles/Makefile2 all
make[1]: Entering directory '/work/08435/srvenkat/frontera/AMGX/build2'
make  -f CMakeFiles/amgx_libs.dir/build.make CMakeFiles/amgx_libs.dir/depend
make[2]: Entering directory '/work/08435/srvenkat/frontera/AMGX/build2'
cd /work/08435/srvenkat/frontera/AMGX/build2 && /opt/apps/cmake/3.24.2/bin/cmake -E cmake_depends "Unix Makefiles" /work/08435/srvenkat/frontera/AMGX /work/08435/srvenkat/frontera/AMGX /work/08435/srvenkat/frontera/AMGX/build2 /work/08435/srvenkat/frontera/AMGX/build2 /work/08435/srvenkat/frontera/AMGX/build2/CMakeFiles/amgx_libs.dir/DependInfo.cmake --color=
make[2]: Leaving directory '/work/08435/srvenkat/frontera/AMGX/build2'
make  -f CMakeFiles/amgx_libs.dir/build.make CMakeFiles/amgx_libs.dir/build
make[2]: Entering directory '/work/08435/srvenkat/frontera/AMGX/build2'
[  0%] Building CUDA object CMakeFiles/amgx_libs.dir/src/aggregation/aggregation_amg_level.cu.o
/opt/apps/cuda/12.0/bin/nvcc -forward-unknown-to-host-compiler -ccbin=mpic++ -DTHRUST_DEVICE_SYSTEM=THRUST_DEVICE_SYSTEM_CUDA -DTHRUST_HOST_SYSTEM=THRUST_HOST_SYSTEM_CPP -I/work/08435/srvenkat/frontera/AMGX/external/rapidjson/include -I/work/08435/srvenkat/frontera/AMGX/src/../include -I/work/08435/srvenkat/frontera/AMGX/thrust/thrust/cmake/../.. -I/work/08435/srvenkat/frontera/AMGX/thrust/dependencies/libcudacxx/include -I/work/08435/srvenkat/frontera/AMGX/thrust/dependencies/cub/cub/cmake/../.. -O3 -DNDEBUG --generate-code=arch=compute_80,code=[compute_80,sm_80] -DNDEBUG --extended-lambda --Werror cross-execution-space-call -DNVTX_RANGES -DDISABLE_MIXED_PRECISION -DCUSPARSE_GENERIC_INTERFACES -DCUSPARSE_USE_GENERIC_SPGEMM -Xcompiler "-qopenmp -DRAPIDJSON_DEFINED -DAMGX_WITH_MPI -rdynamic -fPIC -fvisibility=default" -DTHRUST_CUB_WRAPPED_NAMESPACE=amgx -std=c++14 -MD -MT CMakeFiles/amgx_libs.dir/src/aggregation/aggregation_amg_level.cu.o -MF CMakeFiles/amgx_libs.dir/src/aggregation/aggregation_amg_level.cu.o.d -x cu -c /work/08435/srvenkat/frontera/AMGX/src/aggregation/aggregation_amg_level.cu -o CMakeFiles/amgx_libs.dir/src/aggregation/aggregation_amg_level.cu.o
/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/memory.h(28): error: name followed by "::" must be a class or namespace name

/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/memory.h(29): error: name followed by "::" must be a class or namespace name

/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/memory.h(30): error: name followed by "::" must be a class or namespace name

/work/08435/srvenkat/frontera/AMGX/src/../include/vector_thrust_allocator.h(58): error: qualified name is not allowed

/work/08435/srvenkat/frontera/AMGX/src/../include/vector_thrust_allocator.h(58): error: explicit type is missing ("int" assumed)

/work/08435/srvenkat/frontera/AMGX/src/../include/vector_thrust_allocator.h(58): error: expected a ";"

/work/08435/srvenkat/frontera/AMGX/src/../include/vector_thrust_allocator.h(61): error: qualified name is not allowed

/work/08435/srvenkat/frontera/AMGX/src/../include/vector_thrust_allocator.h(61): error: explicit type is missing ("int" assumed)

/work/08435/srvenkat/frontera/AMGX/src/../include/vector_thrust_allocator.h(61): error: expected a ";"

/work/08435/srvenkat/frontera/AMGX/src/../include/vector_thrust_allocator.h(64): error: qualified name is not allowed

/work/08435/srvenkat/frontera/AMGX/src/../include/vector_thrust_allocator.h(64): error: explicit type is missing ("int" assumed)

/work/08435/srvenkat/frontera/AMGX/src/../include/vector_thrust_allocator.h(64): error: expected a ";"

/work/08435/srvenkat/frontera/AMGX/src/../include/vector_thrust_allocator.h(67): error: qualified name is not allowed

/work/08435/srvenkat/frontera/AMGX/src/../include/vector_thrust_allocator.h(67): error: explicit type is missing ("int" assumed)

/work/08435/srvenkat/frontera/AMGX/src/../include/vector_thrust_allocator.h(67): error: expected a ";"

/work/08435/srvenkat/frontera/AMGX/src/../include/vector_thrust_allocator.h(73): error: a class or namespace qualified name is required       

/work/08435/srvenkat/frontera/AMGX/src/../include/vector_thrust_allocator.h(73): error: global-scope qualifier (leading "::") is not allowed  

/work/08435/srvenkat/frontera/AMGX/src/../include/vector_thrust_allocator.h(73): error: expected a ";"

/work/08435/srvenkat/frontera/AMGX/src/../include/vector_thrust_allocator.h(109): error: identifier "pointer" is undefined

/work/08435/srvenkat/frontera/AMGX/src/../include/vector_thrust_allocator.h(109): error: identifier "reference" is undefined

/work/08435/srvenkat/frontera/AMGX/src/../include/vector_thrust_allocator.h(115): error: identifier "const_pointer" is undefined

/work/08435/srvenkat/frontera/AMGX/src/../include/vector_thrust_allocator.h(115): error: identifier "const_reference" is undefined

/work/08435/srvenkat/frontera/AMGX/src/../include/vector_thrust_allocator.h(115): error: invalid redeclaration of member function "amgx::thrust_amgx_allocator<T, 1>::address(<error-type>)"
(109): here

/work/08435/srvenkat/frontera/AMGX/src/../include/vector_thrust_allocator.h(123): error: identifier "pointer" is undefined

/work/08435/srvenkat/frontera/AMGX/src/../include/vector_thrust_allocator.h(124): error: identifier "const_pointer" is undefined

/work/08435/srvenkat/frontera/AMGX/src/../include/vector_thrust_allocator.h(138): error: identifier "pointer" is undefined

/work/08435/srvenkat/frontera/AMGX/src/../include/vector_thrust_allocator.h(263): error: namespace "amgx::thrust" has no member "device_vector"

/work/08435/srvenkat/frontera/AMGX/src/../include/vector_thrust_allocator.h(263): error: expected a ";"

/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/detail/memory.inl(48): error: name followed by "::" must be a class or namespace name  

/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/detail/memory.inl(49): error: not a class or struct name

/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/detail/memory.inl(63): warning #12-D: parsing restarts here after previous syntax error

Remark: The warnings can be suppressed with "-diag-suppress <warning-number>"

/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/detail/memory.inl(49): error: class or struct definition is missing

/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/detail/memory.inl(63): warning #12-D: parsing restarts here after previous syntax error

/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/memory.h(48): error: class "cusp::CuspMemMap<cusp::host_memory>" has already been defined

/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/memory.h(49): error: class "cusp::CuspMemMap<cusp::host_memory>" has already been defined

/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/array1d.h(54): error: name followed by "::" must be a class or namespace name

/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/array1d.h(54): error: expected an identifier

/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/array1d.h(58): error: name followed by "::" must be a class or namespace name

/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/array1d.h(58): error: expected an identifier

/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/array1d.h(58): warning #1307-D: class member typedef may not be redeclared

/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/array1d.h(58): error: expected a ";"

/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/array1d.h(73): error: a class or namespace qualified name is required

/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/array1d.h(73): error: expected a ">"

/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/array1d.h(77): error: a class or namespace qualified name is required

/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/array1d.h(77): error: expected a ">"

/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/array1d.h(79): error: a class or namespace qualified name is required

/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/array1d.h(79): error: global-scope qualifier (leading "::") is not allowed

/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/array1d.h(79): error: expected a ";"

/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/array1d.h(80): error: a class or namespace qualified name is required

/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/array1d.h(80): error: global-scope qualifier (leading "::") is not allowed

/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/array1d.h(80): error: expected a ";"

/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/array1d.h(84): error: identifier "size_type" is undefined

/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/array1d.h(94): error: identifier "size_type" is undefined

/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/array1d.h(94): error: identifier "value_type" is undefined

/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/array1d.h(98): error: name followed by "::" must be a class or namespace name

/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/array1d.h(98): error: expected an identifier

/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/array1d.h(98): error: expected a ")"

/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/array1d.h(82): error: "Parent" is not a nonstatic data member or base class of class "cusp::array1d<T, MemorySpace>"

/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/array1d.h(85): error: "Parent" is not a nonstatic data member or base class of class "cusp::array1d<T, MemorySpace>"

/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/array1d.h(89): error: name followed by "::" must be a class or namespace name

/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/array1d.h(90): error: name followed by "::" must be a class or namespace name

/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/array1d.h(95): error: "Parent" is not a nonstatic data member or base class of class "cusp::array1d<T, MemorySpace>"

/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/array1d.h(99): error: "Parent" is not a nonstatic data member or base class of class "cusp::array1d<T, MemorySpace>"

/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/array1d.h(103): error: "Parent" is not a nonstatic data member or base class of class "cusp::array1d<T, MemorySpace>"

/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/array1d.h(107): error: name followed by "::" must be a class or namespace name

/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/array1d.h(132): error: namespace "amgx::thrust" has no member "iterator_reference"     

/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/array1d.h(132): error: expected an identifier

/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/array1d.h(133): error: namespace "amgx::thrust" has no member "iterator_difference"    

/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/array1d.h(133): error: expected an identifier

/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/array1d.h(134): error: namespace "amgx::thrust" has no member "iterator_value"

/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/array1d.h(134): error: expected an identifier

/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/array1d.h(136): error: namespace "amgx::thrust" has no member "iterator_system"        

/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/array1d.h(136): error: expected an identifier

/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/array1d.h(140): error: namespace "amgx::thrust" has no member "iterator_pointer"       

/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/array1d.h(140): error: expected an identifier

/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/array1d.h(144): error: identifier "value_type" is undefined

/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/array1d.h(144): error: identifier "memory_space" is undefined

/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/array1d.h(185): error: identifier "reference" is undefined

/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/array1d.h(190): error: identifier "reference" is undefined

/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/array1d.h(195): error: identifier "reference" is undefined

/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/array1d.h(195): error: identifier "difference_type" is undefined

/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/array1d.h(221): error: identifier "pointer" is undefined

/work/08435/srvenkat/frontera/AMGX/src/../include/cusp/detail/array1d.inl(26): error: name followed by "::" must be a class or namespace name 

/work/08435/srvenkat/frontera/AMGX/src/../include/vector.h(166): error: namespace "amgx::thrust" has no member class "host_vector"

/work/08435/srvenkat/frontera/AMGX/src/../include/vector.h(166): error: class or struct definition is missing

/work/08435/srvenkat/frontera/AMGX/src/../include/vector.h(424): warning #12-D: parsing restarts here after previous syntax error

/work/08435/srvenkat/frontera/AMGX/src/../include/vector.h(428): error: not a class or struct name

/work/08435/srvenkat/frontera/AMGX/src/../include/vector.h(624): error: invalid redeclaration of member function "amgx::Vector<amgx::TemplateConfig<(AMGX_MemorySpace)1, t_vecPrec, t_matPrec, t_indPrec>>::operator=(const cusp::array1d<amgx::Vector<amgx::TemplateConfig<(AMGX_MemorySpace)1, t_vecPrec, t_matPrec, t_indPrec>>::value_type, amgx::MemorySpaceMap<(AMGX_MemorySpace)0>::Type> &)"
(623): here

/work/08435/srvenkat/frontera/AMGX/src/../include/vector.h(442): error: "device_vector_alloc" is not a nonstatic data member or base class of class "amgx::Vector<amgx::TemplateConfig<(AMGX_MemorySpace)1, t_vecPrec, t_matPrec, t_indPrec>>"

/work/08435/srvenkat/frontera/AMGX/src/../include/vector.h(443): error: "device_vector_alloc" is not a nonstatic data member or base class of class "amgx::Vector<amgx::TemplateConfig<(AMGX_MemorySpace)1, t_vecPrec, t_matPrec, t_indPrec>>"

/work/08435/srvenkat/frontera/AMGX/src/../include/vector.h(444): error: "device_vector_alloc" is not a nonstatic data member or base class of class "amgx::Vector<amgx::TemplateConfig<(AMGX_MemorySpace)1, t_vecPrec, t_matPrec, t_indPrec>>"

/work/08435/srvenkat/frontera/AMGX/src/../include/vector.h(445): error: "device_vector_alloc" is not a nonstatic data member or base class of class "amgx::Vector<amgx::TemplateConfig<(AMGX_MemorySpace)1, t_vecPrec, t_matPrec, t_indPrec>>"

/work/08435/srvenkat/frontera/AMGX/src/../include/vector.h(446): error: "device_vector_alloc" is not a nonstatic data member or base class of class "amgx::Vector<amgx::TemplateConfig<(AMGX_MemorySpace)1, t_vecPrec, t_matPrec, t_indPrec>>"

/work/08435/srvenkat/frontera/AMGX/src/../include/vector.h(630): error: namespace "amgx::thrust" has no member "raw_pointer_cast"

/work/08435/srvenkat/frontera/AMGX/src/../include/vector.h(635): error: namespace "amgx::thrust" has no member "raw_pointer_cast"

/work/08435/srvenkat/frontera/AMGX/src/../include/distributed/distributed_manager.h(299): error: namespace "amgx::thrust" has no member "host_vector"

/work/08435/srvenkat/frontera/AMGX/src/../include/distributed/distributed_manager.h(299): error: expected a ")"

/work/08435/srvenkat/frontera/AMGX/src/../include/distributed/distributed_manager.h(309): error: identifier "B2L_maps_" is undefined

/work/08435/srvenkat/frontera/AMGX/src/../include/distributed/distributed_manager.h(453): error: namespace "amgx::thrust" has no member "copy"

/work/08435/srvenkat/frontera/AMGX/src/../include/distributed/distributed_manager.h(1129): error: namespace "amgx::thrust" has no member "copy"

/work/08435/srvenkat/frontera/AMGX/src/../include/distributed/distributed_manager.h(1136): error: namespace "amgx::thrust" has no member "copy"

/work/08435/srvenkat/frontera/AMGX/src/../include/distributed/distributed_manager.h(1545): error: namespace "amgx::thrust" has no member "raw_pointer_cast"

/work/08435/srvenkat/frontera/AMGX/src/../include/distributed/distributed_manager.h(1546): error: namespace "amgx::thrust" has no member "raw_pointer_cast"

/work/08435/srvenkat/frontera/AMGX/src/../include/distributed/distributed_manager.h(1615): error: namespace "amgx::thrust" has no member "raw_pointer_cast"

Error limit reached.
100 errors detected in the compilation of "/work/08435/srvenkat/frontera/AMGX/src/aggregation/aggregation_amg_level.cu".
Compilation terminated.
make[2]: *** [CMakeFiles/amgx_libs.dir/build.make:76: CMakeFiles/amgx_libs.dir/src/aggregation/aggregation_amg_level.cu.o] Error 4
make[2]: Leaving directory '/work/08435/srvenkat/frontera/AMGX/build2'
make[1]: *** [CMakeFiles/Makefile2:158: CMakeFiles/amgx_libs.dir/all] Error 2
make[1]: Leaving directory '/work/08435/srvenkat/frontera/AMGX/build2'
make: *** [Makefile:136: all] Error 2

Reproduce the configuration with the shown CMake command and make all VERBOSE=1. Start with the first errors in include/cusp/memory.h and include/vector_thrust_allocator.h, then inspect the related cusp/detail/memory.inl, cusp/array1d.h, and src/aggregation/aggregation_amg_level.cu paths. Done means identifying a compatible build outcome for MVAPICH2-GDR or documenting the confirmed incompatibility.