pyscf / pyscf/gpu4pyscf

Is it possible to manger GPU memory usage in MP2 to prevent crashes?

Open
#620 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Cuda
Stars
351
Forks
84
Avg merge
3d 2h
Merged PRs (30d)
35

Description

Hi guys, I am encountering the issue: when using the A40 GPU for MP2 calculations with the 4C2E integral step, it throws an out-of-memory error. I would like to know if there is a method to manger GPU mempry? trackback as following:

Traceback (most recent call last):
File "/home/user/pyscf_test/test.py", line 64, in
main()
File "/home/user/pyscf_test/test.py", line 44, in main
e_corr, t2 = pt.kernel(with_t2=True)
^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/miniconda3/envs/pyscf/lib/python3.12/site-packages/gpu4pyscf/mp/mp2.py", line 310, in kernel
eris = self.ao2mo(mo_coeff)
^^^^^^^^^^^^^^^^^^^^
File "/home/user/miniconda3/envs/pyscf/lib/python3.12/site-packages/gpu4pyscf/mp/mp2.py", line 338, in ao2mo
return _make_eris(self, mo_coeff, verbose=self.verbose)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/miniconda3/envs/pyscf/lib/python3.12/site-packages/gpu4pyscf/mp/mp2.py", line 152, in _make_eris
eris.ovov = int4c2e.get_int4c2e_ovov(mp.mol, co, cv)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/miniconda3/envs/pyscf/lib/python3.12/site-packages/gpu4pyscf/scf/int4c2e.py", line 349, in get_int4c2e_ovov
ovov += _einsum('oprl,lq->oprq', int4c_ovoa, orbv[l0:l1])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/miniconda3/envs/pyscf/lib/python3.12/site-packages/cupy/linalg/_einsum.py", line 673, in einsum
arr_out, sub_out = reduced_binary_einsum(
^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/miniconda3/envs/pyscf/lib/python3.12/site-packages/cupy/linalg/_einsum.py", line 418, in reduced_binary_einsum
arr_out = cupy.matmul(tmp0, tmp1).reshape(shapes_out)
^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/miniconda3/envs/pyscf/lib/python3.12/site-packages/cupy/_core/_gufuncs.py", line 669, in call
outs = cupy.empty(out_shape, dtype=ret_dtype, order=filter_order)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/miniconda3/envs/pyscf/lib/python3.12/site-packages/cupy/_creation/basic.py", line 32, in empty
return cupy.ndarray(shape, dtype, order=order)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "cupy/_core/core.pyx", line 167, in cupy._core.core.ndarray.new
File "cupy/_core/core.pyx", line 254, in cupy._core.core._ndarray_base._init
File "cupy/cuda/memory.pyx", line 875, in cupy.cuda.memory.alloc
File "/home/user/miniconda3/envs/pyscf/lib/python3.12/site-packages/gpu4pyscf/lib/cupy_helper.py", line 1150, in malloc
return cuda_malloc(size)
^^^^^^^^^^^^^^^^^
File "cupy/cuda/memory.pyx", line 737, in cupy.cuda.memory._malloc
File "cupy/cuda/memory.pyx", line 738, in cupy.cuda.memory._malloc
File "cupy/cuda/memory.pyx", line 112, in cupy.cuda.memory.Memory.init
File "cupy_backends/cuda/api/runtime.pyx", line 528, in cupy_backends.cuda.api.runtime.malloc
File "cupy_backends/cuda/api/runtime.pyx", line 146, in cupy_backends.cuda.api.runtime.check_status
cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorMemoryAllocation: out of memory

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the failure from test.py at pt.kernel(with_t2=True), then inspect gpu4pyscf/mp/mp2.py and scf/int4c2e.py around _make_eris and get_int4c2e_ovov. Check cupy_helper.py's allocation path and determine whether the project has an established memory-management approach. Done should be a documented or tested resolution to the reported A40 out-of-memory failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
hpc, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.