Performance regression on H200 GPUs in v1.5.2 compared to v1.4.3
Nobody has claimed this yet.
- Dominant language
- Cuda
- Stars
- 351
- Forks
- 84
- Avg merge
- 3d 2h
- Merged PRs (30d)
- 35
Description
Observed significant performance degradation when running GPU4PySCF v1.5.2 on NVIDIA H200 GPUs. The v1.4.3 precompiled package performs much better on the same H200 hardware. Also, the performance on H200 GPUs is significantly slower than on V100S GPUs.
Environment
- GPU:
NVIDIA H200: CUDA 12.8, Driver 570.195.03, Compute capability 9.0
NVIDIA V100S: CUDA 13.0, Driver 580.95.05, Compute capability 7.0 - CuBLAS: 12.8.3.14 (H200) / 12.6.4.1 (V100S)
- CuPy: 13.6.0 / 13.4.1
Test case: Structure optimization of ethanol (CCO) using B3LYPG/6-311G* with geomeTRIC; both runs complete in 7 optimization steps.
Additional observations:
- The prebuilt v1.5.2 package has very slow imports and warmup on H200 (~3 minutes total). Recompiling with CUDA_ARCH="90" significantly improves this.
- However, even after recompiling, veff calculation is still slower than v1.4.3 on H200.
- _make_tril_pair_mappings (v1.5.2) takes ~1.3s on H200 vs ~0.03s on V100S at first call
- The nested loop in get_k() (for task in tasks: with inner lib.prange) launches many small kernels
Root cause analysis:
- The v1.5.2 get_k() function uses _make_tril_pair_mappings with many small CuPy operations in nested loops, and launches multiple kernels per task. H200 GPUs have higher kernel launch latency, so the many small operations may be more heavily penalized.
- v1.4.3 uses _make_tril_tile_mappings and a simpler loop structure with fewer kernel launches.
Suggested fix:
Consider adding a code path for newer GPU architectures (e.g., H200/Hopper) that reduces the number of kernel launches, possibly reverting to the v1.4.3 tile-based approach for these GPUs.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by comparing get_k() and _make_tril_pair_mappings in v1.5.2 with the v1.4.3 tile-based path, focusing on the nested task loop and kernel launches. Reproduce the ethanol B3LYPG/6-311G* optimization on an H200 and compare veff timing, first-call mapping time, imports, and warmup against v1.4.3 and V100S. Done means the H200 path avoids the reported launch overhead without regressing the existing comparison.
Written by the indexing model from the issue text.
Assessment
- Domain
- performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100