pyscf / pyscf/gpu4pyscf

Performance regression on H200 GPUs in v1.5.2 compared to v1.4.3

Open
#649 1 comment 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Cuda
Stars
351
Forks
84
Avg merge
3d 2h
Merged PRs (30d)
35

Description

Observed significant performance degradation when running GPU4PySCF v1.5.2 on NVIDIA H200 GPUs. The v1.4.3 precompiled package performs much better on the same H200 hardware. Also, the performance on H200 GPUs is significantly slower than on V100S GPUs.

Environment

  • GPU:

    NVIDIA H200: CUDA 12.8, Driver 570.195.03, Compute capability 9.0
    NVIDIA V100S: CUDA 13.0, Driver 580.95.05, Compute capability 7.0

  • CuBLAS: 12.8.3.14 (H200) / 12.6.4.1 (V100S)
  • CuPy: 13.6.0 / 13.4.1

Test case: Structure optimization of ethanol (CCO) using B3LYPG/6-311G* with geomeTRIC; both runs complete in 7 optimization steps.

Image

Additional observations:

  • The prebuilt v1.5.2 package has very slow imports and warmup on H200 (~3 minutes total). Recompiling with CUDA_ARCH="90" significantly improves this.
  • However, even after recompiling, veff calculation is still slower than v1.4.3 on H200.
  • _make_tril_pair_mappings (v1.5.2) takes ~1.3s on H200 vs ~0.03s on V100S at first call
  • The nested loop in get_k() (for task in tasks: with inner lib.prange) launches many small kernels

Root cause analysis:

  • The v1.5.2 get_k() function uses _make_tril_pair_mappings with many small CuPy operations in nested loops, and launches multiple kernels per task. H200 GPUs have higher kernel launch latency, so the many small operations may be more heavily penalized.
  • v1.4.3 uses _make_tril_tile_mappings and a simpler loop structure with fewer kernel launches.

Suggested fix:
Consider adding a code path for newer GPU architectures (e.g., H200/Hopper) that reduces the number of kernel launches, possibly reverting to the v1.4.3 tile-based approach for these GPUs.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by comparing get_k() and _make_tril_pair_mappings in v1.5.2 with the v1.4.3 tile-based path, focusing on the nested task loop and kernel launches. Reproduce the ethanol B3LYPG/6-311G* optimization on an H200 and compare veff timing, first-call mapping time, imports, and warmup against v1.4.3 and V100S. Done means the H200 path avoids the reported launch overhead without regressing the existing comparison.

Written by the indexing model from the issue text.

Assessment

Domain
performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.