"Cannot allocate pinned memory" error on a supercomputer

I am encountering a "Cannot allocate pinned memory" error while running a program that uses AMGX solvers on a supercomputer that uses the SLURM Workload Manager. The program fails to allocate the necessary pinned memory for efficient GPU memory transfers.

Here's the full output file:

Nonlinear Elasticity FEM Solver
Updated Lagrangian Formulation
Setting material and elasticity type
Setting boundary conditions
Setup of non-linear elasticity problem
Reading parameters from XML file
Creating Incompressible Material
Bulk modulus: 300
 Hyperelastic material: Guccione
 Material properties: 10 1 1 1 0 300
 Number of nodal loads: 0
 Number of prescribed displ.: 249
 Number of traction (Neumann) loads: 0
 Number of dirichlet boundary conds:0
 Number of normal pressure loads: 1
 Number of spring boundary conds: 0
Solving problem
Reading XML mesh
Reading XML mesh file: ./prob2_12x27x2_k300.xml
Fiber model: fiber_isotropic
Mesh information
 Number of dimensions: 3
 Number of nodes: 975
 Number of elements: 648
 Number of boundary elements: 324
Output step: 1
Solving nonlinear problem (UL) using NonlinearSolver
Initial inner volume: 3189.27
Initial cavity volume: 0
Size of the problem 2925
Matrices and vectors creation done
 Load increment 1
 1 Newton-LS step
Caught amgx exception: Cannot allocate pinned memory
 at: /prj/hearttwins/jonatas.costa/source/amgx/src/global_thread_handle.cu:351
Stack trace:
 /scratch/hearttwins/jonatas.costa/source/amgx/lib/libamgxsh.so : amgx::memory::PinnedMemoryPool::PinnedMemoryPool()+0x412
 /scratch/hearttwins/jonatas.costa/source/amgx/lib/libamgxsh.so : amgx::allocate_resources(unsigned long, unsigned long, unsigned long, unsigned long, unsigned long)+0x3b
 /scratch/hearttwins/jonatas.costa/source/amgx/lib/libamgxsh.so : amgx::Resources::Resources(amgx::AMG_Configuration*, void*, int, int const*)+0xc24
 /scratch/hearttwins/jonatas.costa/source/amgx/lib/libamgxsh.so : AMGX_resources_create_simple()+0x8b
 ./nonlinearelas : petsc::LinearSolver::solve(petsc::Matrix&, petsc::Vector&, petsc::Vector&, double)+0x25d
 ./nonlinearelas : NewtonLineSearch::solve()+0x222
 ./nonlinearelas : UpdatedLagrangian::solve()+0x3c4
 ./nonlinearelas : NonlinearElasticity::run()+0x6b
 ./nonlinearelas : main()+0x479
 /lib64/libc.so.6 : __libc_start_main()+0xf5
 ./nonlinearelas() [0x45336f]

Here's the SLURM output:

AMGX ERROR: file /prj/hearttwins/jonatas.costa/nodal_cardiax/src/linalg/petsc_linear_solver.cpp line    371
AMGX ERROR: Insufficient memory.
application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0
[unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1
:
system msg for write_line failure : Bad file descriptor

And those were the SLURM options utilized in this test:

#SBATCH --nodes=1             
#SBATCH --ntasks-per-node=1
#SBATCH --ntasks=1
#SBATCH -p nvidia_dev
#SBATCH --time=0:01:00
#SBATCH -J cardiax_test
#SBATCH --gres=gpu:0
#SBATCH --mem-bind=local

System info

Operating System: Linux Red Hat 7.9
CUDA Version: 12.3
GCC Version: 9.3
MPI Version: 3.4
AMGX Version: 2.5.0
GPU Model: NVIDIA Tesla K40t
NVIDIA Driver Version: 470.82.01

Any guidance or suggestions on resolving this issue would be greatly appreciated. Thank you!

Start at nodal_cardiax/src/linalg/petsc_linear_solver.cpp line 371 and trace the AMGX_resources_create_simple() call through the reported PinnedMemoryPool failure. Reproduce the run with the provided SLURM options and system details, then determine whether the allocation succeeds under a corrected configuration or a documented limitation.