[Build]How to use OpenMPI and multi GPU parallel calling AMGX?

Describe the issue

I can obtain the correct results when using a single GPU to call AMGX to solve a system of linear equations (Poisson's equations), but when using openmpi and multi GPU parallel calls to AMGX, an error occurs. This error may be caused by an error in uploading the matrix when calling AMGX_matrix_upload_all_global, or by parameter errors when creating configuration files or solvers. How should I give the relevant parameters of AMGX_matrix_upload_all_global when calling AMGX with multiple GPUs? Especially how is the ghost grid set up? What are the requirements for matrices when setting up ghost grids? What are the precautions when establishing configuration files, solvers, parallel environments, etc?

A clear and concise description of what the issue is.

Environment information:

• OS: [Ubuntu 20.04]
• Compiler version: [e.g. mpicxx]
• CMake version: [3.23]
• CUDA used for AMGX compilation: [e.g. CUDA12.2]
• MPI version (if applicable): [e.g. OpenMPI 4.0.3]
• AMGX version or commit hash [e.g. v2.3.0]

Compilation information
mpicxx -cuda -gpu=ccall,cuda12.2 CSR_3Dplan_global.cu -L /home/zcy/software/AMGX-main-nvhpc/lib -lamgxsh -I /home/zcy/software/AMGX-main-nvhpc/include/ -L /opt/nvidia/hpc_sdk/Linux_x86_64/23.7/comm_libs/mpi/lib/ -lmpi -lmpi_cxx -L /opt/nvidia/hpc_sdk/Linux_x86_64/23.7/math_libs/12.2/lib64 -lcufft -I /opt/nvidia/hpc_sdk/Linux_x86_64/23.7/math_libs/12.2/include

Issue information

at: /home/stu1/software/AMGX-main-nvhpc/src/distributed/comms_visitors3.cu:23
Stack trace:
/home/stu1/software/AMGX-main-nvhpc/lib/libamgxsh.so : ()+0x21eb697
/home/stu1/software/AMGX-main-nvhpc/lib/libamgxsh.so : amgx::ExcHalo2AsyncFunctor<amgx::TemplateConfig<(AMGX_MemorySpace)1, (AMGX_VecPrecision)0, (AMGX_MatPrecision)0, (AMGX_IndPrecision)2>, amgx::Vector<amgx::TemplateConfig<(AMGX_MemorySpace)1, (AMGX_VecPrecision)0, (AMGX_MatPrecision)0, (AMGX_IndPrecision)2> > >::operator()(amgx::CommsMPIHostBufferStream<amgx::TemplateConfig<(AMGX_MemorySpace)1, (AMGX_VecPrecision)0, (AMGX_MatPrecision)0, (AMGX_IndPrecision)2> >&)+0
/home/stu1/software/AMGX-main-nvhpc/lib/libamgxsh.so : void amgx::CommsMPIHostBufferStream<amgx::TemplateConfig<(AMGX_MemorySpace)1, (AMGX_VecPrecision)0, (AMGX_MatPrecision)0, (AMGX_IndPrecision)2> >::do_exchange_halo<amgx::Vector<amgx::TemplateConfig<(AMGX_MemorySpace)1, (AMGX_VecPrecision)0, (AMGX_MatPrecision)0, (AMGX_IndPrecision)2> > >(amgx::Vector<amgx::TemplateConfig<(AMGX_MemorySpace)1, (AMGX_VecPrecision)0, (AMGX_MatPrecision)0, (AMGX_IndPrecision)2> >&, amgx::Matrix<amgx::TemplateConfig<(AMGX_MemorySpace)1, (AMGX_VecPrecision)0, (AMGX_MatPrecision)0, (AMGX_IndPrecision)2> > const&, int)+0x206
/home/stu1/software/AMGX-main-nvhpc/lib/libamgxsh.so : void amgx::multiply<amgx::TemplateConfig<(AMGX_MemorySpace)1, (AMGX_VecPrecision)0, (AMGX_MatPrecision)0, (AMGX_IndPrecision)2> >(amgx::Matrix<amgx::TemplateConfig<(AMGX_MemorySpace)1, (AMGX_VecPrecision)0, (AMGX_MatPrecision)0, (AMGX_IndPrecision)2> >&, amgx::Vector<amgx::TemplateConfig<(AMGX_MemorySpace)1, (AMGX_VecPrecision)0, (AMGX_MatPrecision)0, (AMGX_IndPrecision)2> >&, amgx::Vector<amgx::TemplateConfig<(AMGX_MemorySpace)1, (AMGX_VecPrecision)0, (AMGX_MatPrecision)0, (AMGX_IndPrecision)2> >&, amgx::ViewType)+0xf45
/home/stu1/software/AMGX-main-nvhpc/lib/libamgxsh.so : void amgx::axmb<amgx::Operator<amgx::TemplateConfig<(AMGX_MemorySpace)1, (AMGX_VecPrecision)0, (AMGX_MatPrecision)0, (AMGX_IndPrecision)2> >, amgx::Vector<amgx::TemplateConfig<(AMGX_MemorySpace)1, (AMGX_VecPrecision)0, (AMGX_MatPrecision)0, (AMGX_IndPrecision)2> > >(amgx::Operator<amgx::TemplateConfig<(AMGX_MemorySpace)1, (AMGX_VecPrecision)0, (AMGX_MatPrecision)0, (AMGX_IndPrecision)2> >&, amgx::Vector<amgx::TemplateConfig<(AMGX_MemorySpace)1, (AMGX_VecPrecision)0, (AMGX_MatPrecision)0, (AMGX_IndPrecision)2> >&, amgx::Vector<amgx::TemplateConfig<(AMGX_MemorySpace)1, (AMGX_VecPrecision)0, (AMGX_MatPrecision)0, (AMGX_IndPrecision)2> >&, amgx::Vector<amgx::TemplateConfig<(AMGX_MemorySpace)1, (AMGX_VecPrecision)0, (AMGX_MatPrecision)0, (AMGX_IndPrecision)2> >&, int, int)+0x58
/home/stu1/software/AMGX-main-nvhpc/lib/libamgxsh.so : amgx::FGMRES_Solver<amgx::TemplateConfig<(AMGX_MemorySpace)1, (AMGX_VecPrecision)0, (AMGX_MatPrecision)0, (AMGX_IndPrecision)2> >::solve_iteration(amgx::Vector<amgx::TemplateConfig<(AMGX_MemorySpace)1, (AMGX_VecPrecision)0, (AMGX_MatPrecision)0, (AMGX_IndPrecision)2> >&, amgx::Vector<amgx::TemplateConfig<(AMGX_MemorySpace)1, (AMGX_VecPrecision)0, (AMGX_MatPrecision)0, (AMGX_IndPrecision)2> >&, bool)+0x9e8
/home/stu1/software/AMGX-main-nvhpc/lib/libamgxsh.so : amgx::Solver<amgx::TemplateConfig<(AMGX_MemorySpace)1, (AMGX_VecPrecision)0, (AMGX_MatPrecision)0, (AMGX_IndPrecision)2> >::solve(amgx::Vector<amgx::TemplateConfig<(AMGX_MemorySpace)1, (AMGX_VecPrecision)0, (AMGX_MatPrecision)0, (AMGX_IndPrecision)2> >&, amgx::Vector<amgx::TemplateConfig<(AMGX_MemorySpace)1, (AMGX_VecPrecision)0, (AMGX_MatPrecision)0, (AMGX_IndPrecision)2> >&, bool)+0x594
/home/stu1/software/AMGX-main-nvhpc/lib/libamgxsh.so : amgx::Solver<amgx::TemplateConfig<(AMGX_MemorySpace)1, (AMGX_VecPrecision)0, (AMGX_MatPrecision)0, (AMGX_IndPrecision)2> >::solve_no_throw(amgx::Vector<amgx::TemplateConfig<(AMGX_MemorySpace)1, (AMGX_VecPrecision)0, (AMGX_MatPrecision)0, (AMGX_IndPrecision)2> >&, amgx::Vector<amgx::TemplateConfig<(AMGX_MemorySpace)1, (AMGXCaught amgx exception: Vector size too small: not enough space for halo elements.
Vector: {tag = 1, size = 288}
Required size: 304

Looking forward to your answer! Best wishes for you!

Start with AMGX_matrix_upload_all_global and the multi-GPU ghost-grid requirements described in the issue. Inspect src/distributed/comms_visitors3.cu at the reported stack-trace location and review the MPI compilation command and Vector size error. Done means identifying the incorrect matrix, halo, configuration, or parallel setup and demonstrating a working multi-GPU call.