amgx_mpi_poisson5pt and amgx_mpi_poisson7

Hi,
I just installed AmgX on Piz Daint and I need to run the Laplace examples both in 2d and in 3d.

I am not able to run the code on more nodes by using increasing dimensions.
For example, if I run the following command:
srun -C gpu -n 4 examples/amgx_mpi_poisson5pt -p 256 256 -c ../core/configs/PCG_AGGREGATION_JACOBI.json

for using a fixed grid of 256x256 grid nodes per each MPI tasks, requiring 4 MPI tasks I have the following error:

srun: job 26918173 queued and waiting for resources
srun: job 26918173 has been allocated resources
Process 0 selecting device 0
AMGX version 2.1.0.131-opensource
Built on Nov 12 2020, 17:58:51
Compiled with CUDA Runtime 10.2, using CUDA driver 10.2
Warning: No mode specified, using dDDI by default.
Cannot read file as JSON object, trying as AMGX config
Converting config string to current config version
Parsing configuration string: exception_handling=1 ;
Caught amgx exception: Cannot allocate pinned memory
at: /users/pdambra/AMGX/base/src/global_thread_handle.cu:374
Stack trace:
/users/pdambra/AMGX/build/libamgxsh.so : amgx::memory::PinnedMemoryPool::PinnedMemoryPool()+0xef
/users/pdambra/AMGX/build/libamgxsh.so : amgx::allocate_resources(unsigned long, unsigned long, unsigned long, unsigned long, unsigned long)+0x3b
/users/pdambra/AMGX/build/libamgxsh.so : amgx::Resources::Resources(amgx::AMG_Configuration*, void*, int, int const*)+0xb5e
/users/pdambra/AMGX/build/libamgxsh.so : AMGX_resources_create()+0xa4
examples/amgx_mpi_poisson5pt() [0x402835]
/lib64/libc.so.6 : __libc_start_main()+0xea
examples/amgx_mpi_poisson5pt() [0x4035ca]

Caught signal 11 - SIGSEGV (segmentation violation)
Process 1 selecting device 0
Warning: No mode specified, using dDDI by default.
Caught amgx exception: Cannot allocate pinned memory
at: /users/pdambra/AMGX/base/src/global_thread_handle.cu:374
Stack trace:
/users/pdambra/AMGX/build/libamgxsh.so : amgx::memory::PinnedMemoryPool::PinnedMemoryPool()+0xef
/users/pdambra/AMGX/build/libamgxsh.so : amgx::allocate_resources(unsigned long, unsigned long, unsigned long, unsigned long, unsigned long)+0x3b
/users/pdambra/AMGX/build/libamgxsh.so : amgx::Resources::Resources(amgx::AMG_Configuration*, void*, int, int const*)+0xb5e
/users/pdambra/AMGX/build/libamgxsh.so : AMGX_resources_create()+0xa4
examples/amgx_mpi_poisson5pt() [0x402835]
/lib64/libc.so.6 : __libc_start_main()+0xea
examples/amgx_mpi_poisson5pt() [0x4035ca]

/users/pdambra/AMGX/build/libamgxsh.so : amgx::handle_signals(int)+0x9a
/apps/daint/UES/xalt/xalt2/software/xalt/2.8.10/lib64/libpthread.so.0 : ()+0x132d0
/users/pdambra/AMGX/build/libamgxsh.so : amgx::CWrapHandle<AMGX_resources_handle_struct*, amgx::Resources>::CWrapHandle(AMGX_resources_handle_struct*)+0x37
/users/pdambra/AMGX/build/libamgxsh.so : AMGX_matrix_create_impl()+0x46
/users/pdambra/AMGX/build/libamgxsh.so : AMGX_matrix_create()+0x3a
examples/amgx_mpi_poisson5pt() [0x40284b]
/lib64/libc.so.6 : __libc_start_main()+0xea
examples/amgx_mpi_poisson5pt() [0x4035ca]
/users/pdambra/AMGX/build/libamgxsh.so : amgx::handle_signals(int)+0x9a
/apps/daint/UES/xalt/xalt2/software/xalt/2.8.10/lib64/libpthread.so.0 : ()+0x132d0
/users/pdambra/AMGX/build/libamgxsh.so : amgx::CWrapHandle<AMGX_resources_handle_struct*, amgx::Resources>::CWrapHandle(AMGX_resources_handle_struct*)+0x37
/users/pdambra/AMGX/build/libamgxsh.so : AMGX_matrix_create_impl()+0x46
/users/pdambra/AMGX/build/libamgxsh.so : AMGX_matrix_create()+0x3a
examples/amgx_mpi_poisson5pt() [0x40284b]
/lib64/libc.so.6 : __libc_start_main()+0xea
examples/amgx_mpi_poisson5pt() [0x4035ca]
Process 2 selecting device 0
Warning: No mode specified, using dDDI by default.
Caught amgx exception: Cannot allocate pinned memory
at: /users/pdambra/AMGX/base/src/global_thread_handle.cu:374
Stack trace:
/users/pdambra/AMGX/build/libamgxsh.so : amgx::memory::PinnedMemoryPool::PinnedMemoryPool()+0xef
/users/pdambra/AMGX/build/libamgxsh.so : amgx::allocate_resources(unsigned long, unsigned long, unsigned long, unsigned long, unsigned long)+0x3b
/users/pdambra/AMGX/build/libamgxsh.so : amgx::Resources::Resources(amgx::AMG_Configuration*, void*, int, int const*)+0xb5e
/users/pdambra/AMGX/build/libamgxsh.so : AMGX_resources_create()+0xa4
examples/amgx_mpi_poisson5pt() [0x402835]
/lib64/libc.so.6 : __libc_start_main()+0xea
examples/amgx_mpi_poisson5pt() [0x4035ca]

/users/pdambra/AMGX/build/libamgxsh.so : amgx::handle_signals(int)+0x9a
/apps/daint/UES/xalt/xalt2/software/xalt/2.8.10/lib64/libpthread.so.0 : ()+0x132d0
/users/pdambra/AMGX/build/libamgxsh.so : amgx::CWrapHandle<AMGX_resources_handle_struct*, amgx::Resources>::CWrapHandle(AMGX_resources_handle_struct*)+0x37
/users/pdambra/AMGX/build/libamgxsh.so : AMGX_matrix_create_impl()+0x46
/users/pdambra/AMGX/build/libamgxsh.so : AMGX_matrix_create()+0x3a
examples/amgx_mpi_poisson5pt() [0x40284b]
/lib64/libc.so.6 : __libc_start_main()+0xea
examples/amgx_mpi_poisson5pt() [0x4035ca]
srun: error: nid02525: tasks 0,2: Segmentation fault
srun: Terminating job step 26918173.0
slurmstepd: error: *** STEP 26918173.0 ON nid02525 CANCELLED AT 2020-11-13T11:26:37 ***
srun: error: nid02525: task 3: Terminated
srun: error: nid02525: task 1: Segmentation fault (core dumped)
srun: Force Terminated job step 26918173.0

Similar errors occur for the 3d example:

srun -C gpu -N 2 examples/amgx_mpi_poisson7 -p 64 64 64 -c ../core/configs/PCG_AGGREGATION_JACOBI.json
srun: job 26918336 queued and waiting for resources

srun: job 26918336 has been allocated resources
Process 0 selecting device 0
AMGX version 2.1.0.131-opensource
Built on Nov 12 2020, 17:58:51
Compiled with CUDA Runtime 10.2, using CUDA driver 10.2
Warning: No mode specified, using dDDI by default.
Cannot read file as JSON object, trying as AMGX config
Converting config string to current config version
Parsing configuration string: exception_handling=1 ;
Caught signal 11 - SIGSEGV (segmentation violation)
/users/pdambra/AMGX/build/libamgxsh.so : amgx::handle_signals(int)+0x9a
/apps/daint/UES/xalt/xalt2/software/xalt/2.8.10/lib64/libpthread.so.0 : ()+0x132d0
/lib64/libc.so.6 : ()+0x3deb0
examples/amgx_mpi_poisson7() [0x4029ab]
/lib64/libc.so.6 : __libc_start_main()+0xea
examples/amgx_mpi_poisson7() [0x4031aa]
Process 1 selecting device 0
Warning: No mode specified, using dDDI by default.
/users/pdambra/AMGX/build/libamgxsh.so : amgx::handle_signals(int)+0x9a
/apps/daint/UES/xalt/xalt2/software/xalt/2.8.10/lib64/libpthread.so.0 : ()+0x132d0
/lib64/libc.so.6 : ()+0x3deb0
examples/amgx_mpi_poisson7() [0x4029ab]
/lib64/libc.so.6 : __libc_start_main()+0xea
examples/amgx_mpi_poisson7() [0x4031aa]

Could you help me?
Thanks

Start by reproducing the two commands for amgx_mpi_poisson5pt and amgx_mpi_poisson7 with PCG_AGGREGATION_JACOBI.json, then inspect base/src/global_thread_handle.cu around the reported PinnedMemoryPool failure and the example entry points. Done means identifying the cause of the multi-process failure and confirming that both 2D and 3D multi-node runs no longer terminate with the reported allocation error and segmentation fault.