QMCPACK / QMCPACK/qmcpack

Out of memory issue on Polaris due to CUDA pinned memory on Polaris

Open
#5,557 5 comments 0 reactions 0 assignees View on GitHub
batched_bug bug stale
Dominant language
C++
Stars
403
Forks
154
Avg merge
1d 12h
Merged PRs (30d)
82

Description

**Describe the bug**
Runs stopped with the following error
```
cudaAssert: cudaErrorMemoryAllocation out of memory, file /home/yeluo/opt/qmcpack/src/Platforms/CUDA/MemManageCUDA.hpp, line 74
```
when calling cudaHostRegister. However, host memory usage is way below available DDR capacity.

**To Reproduce**
Steps to reproduce the behavior:
1. all code release with DiracDeterminantBatched
2. NiO performance benchmark a64 with 2048 walkers per rank.
3. Running 2-4 ranks per node fails. 1 MPI rank works
4. Each MPI rank sees all 4 GPUs.

**Expected behavior**
The simulation should run with 1-4 ranks.

**System:**
ALCF Polaris

**Additional context**
I injected counters to count the peak amount of registered host memory (pinned memory) segments.
1 MPI rank run completed with max at ~34k
2 MPI rank run hit error at max ~32k per rank
3 MPI rank run hit error at max ~21k per rank
4 MPI rank run hit error at max ~16k per rank

a. There is a cap at around 65536 magic number. My guess `vm.max_map_count=65530`
b. It seems MPI (Cray MPICH) related. Likely due to the notorious XPMEM.
c. [workaround](https://docs.alcf.anl.gov/polaris/running-jobs/using-gpus/#binding-mpi-ranks-to-gpus) exposing one GPU per rank made all cases to run.

Long term solution from our side. We need to to bulk allocation/registration and views instead of doing that per walker.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.