Is it possible to manger GPU memory usage in MP2 to prevent crashes?
Nobody has claimed this yet.
- Dominant language
- Cuda
- Stars
- 351
- Forks
- 84
- Avg merge
- 3d 2h
- Merged PRs (30d)
- 35
Description
Hi guys, I am encountering the issue: when using the A40 GPU for MP2 calculations with the 4C2E integral step, it throws an out-of-memory error. I would like to know if there is a method to manger GPU mempry? trackback as following:
Traceback (most recent call last):
File "/home/user/pyscf_test/test.py", line 64, in
main()
File "/home/user/pyscf_test/test.py", line 44, in main
e_corr, t2 = pt.kernel(with_t2=True)
^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/miniconda3/envs/pyscf/lib/python3.12/site-packages/gpu4pyscf/mp/mp2.py", line 310, in kernel
eris = self.ao2mo(mo_coeff)
^^^^^^^^^^^^^^^^^^^^
File "/home/user/miniconda3/envs/pyscf/lib/python3.12/site-packages/gpu4pyscf/mp/mp2.py", line 338, in ao2mo
return _make_eris(self, mo_coeff, verbose=self.verbose)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/miniconda3/envs/pyscf/lib/python3.12/site-packages/gpu4pyscf/mp/mp2.py", line 152, in _make_eris
eris.ovov = int4c2e.get_int4c2e_ovov(mp.mol, co, cv)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/miniconda3/envs/pyscf/lib/python3.12/site-packages/gpu4pyscf/scf/int4c2e.py", line 349, in get_int4c2e_ovov
ovov += _einsum('oprl,lq->oprq', int4c_ovoa, orbv[l0:l1])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/miniconda3/envs/pyscf/lib/python3.12/site-packages/cupy/linalg/_einsum.py", line 673, in einsum
arr_out, sub_out = reduced_binary_einsum(
^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/miniconda3/envs/pyscf/lib/python3.12/site-packages/cupy/linalg/_einsum.py", line 418, in reduced_binary_einsum
arr_out = cupy.matmul(tmp0, tmp1).reshape(shapes_out)
^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/miniconda3/envs/pyscf/lib/python3.12/site-packages/cupy/_core/_gufuncs.py", line 669, in call
outs = cupy.empty(out_shape, dtype=ret_dtype, order=filter_order)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/miniconda3/envs/pyscf/lib/python3.12/site-packages/cupy/_creation/basic.py", line 32, in empty
return cupy.ndarray(shape, dtype, order=order)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "cupy/_core/core.pyx", line 167, in cupy._core.core.ndarray.new
File "cupy/_core/core.pyx", line 254, in cupy._core.core._ndarray_base._init
File "cupy/cuda/memory.pyx", line 875, in cupy.cuda.memory.alloc
File "/home/user/miniconda3/envs/pyscf/lib/python3.12/site-packages/gpu4pyscf/lib/cupy_helper.py", line 1150, in malloc
return cuda_malloc(size)
^^^^^^^^^^^^^^^^^
File "cupy/cuda/memory.pyx", line 737, in cupy.cuda.memory._malloc
File "cupy/cuda/memory.pyx", line 738, in cupy.cuda.memory._malloc
File "cupy/cuda/memory.pyx", line 112, in cupy.cuda.memory.Memory.init
File "cupy_backends/cuda/api/runtime.pyx", line 528, in cupy_backends.cuda.api.runtime.malloc
File "cupy_backends/cuda/api/runtime.pyx", line 146, in cupy_backends.cuda.api.runtime.check_status
cupy_backends.cuda.api.runtime.CUDARuntimeError: cudaErrorMemoryAllocation: out of memory
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the failure from test.py at pt.kernel(with_t2=True), then inspect gpu4pyscf/mp/mp2.py and scf/int4c2e.py around _make_eris and get_int4c2e_ovov. Check cupy_helper.py's allocation path and determine whether the project has an established memory-management approach. Done should be a documented or tested resolution to the reported A40 out-of-memory failure.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- hpc, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100