QMCPACK Issue Draft: Complex CUDA/NCCL AFQMC Factorized SparseTensor Crash
- Dominant language
- C++
- Stars
- 403
- Forks
- 154
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 90
Description
## Issue Body
### Summary
I hit a reproducible failure in QMCPACK 4.3.0 when running a complex-valued AFQMC calculation with an HDF5 `Factorized` Hamiltonian on the CUDA/NCCL build.
The same small complex AFQMC input runs with the complex CPU/PHDF5 build. The complex GPU/NCCL build also runs an HDF5 THC Hamiltonian input. The failing path appears specific to the complex CUDA/HIP `SparseTensor` / Factorized Hamiltonian path.
I have a local workaround patch that makes the one-rank complex GPU/NCCL Factorized/Cholesky smoke test complete successfully. I am opening this issue first to confirm whether the maintainers prefer this conservative host-scratch fallback, or a proper device sparse implementation.
### Environment
- QMCPACK: 4.3.0
- Upstream commit tested: `bb7eede051f98ec03296664b304982e655f960c4`
- Build mode:
- `QMC_COMPLEX=ON`
- `BUILD_AFQMC=ON`
- `ENABLE_PHDF5=ON`
- `ENABLE_CUDA=ON`
- `BUILD_AFQMC_WITH_NCCL=ON`
- GPU: NVIDIA A100-SXM4-80GB
- CUDA: 12.9
- NCCL: 2.25
- MPI: OpenMPI via Conda environment
### Minimal Run Shape
Input shape:
```xml
26
4
4
hdf5
../choldump.h5
ascii
../wfn.dat
closed
0.01
1
1
2
```
Command:
```bash
CUDA_VISIBLE_DEVICES=0 OMP_NUM_THREADS=1 \
mpiexec -n 1 --bind-to none \
/path/to/qmcpack qmc_short_chol.in.xml --gpu
```
### Observed Behavior
Before the local patch, the run failed in the complex CUDA/NCCL build during AFQMC propagator initialization / mean-field subtraction.
The crash mapped into the `SparseTensor` Factorized path, including:
- `AFQMC/HamiltonianOperations/SparseTensor.hpp`
- `SparseTensor::vbias`
- `SparseTensor::vHS`
- `NOMSD::vMF`
- `PropagatorFactory::buildAFQMCPropagator`
After patching only `vHS` and `vbias`, the code progressed further but then failed in the energy path with a host/device GEMV mismatch:
```text
types: (gemv) T: std::complex
ptrA: device::device_pointer>
ptrB: const std::complex*
ptrC: device::device_pointer>
Error: Calling qmc_cuda::gemv catch all.
```
This suggests that the Factorized `SparseTensor` path is passing CUDA device dense arrays into host sparse/dense operations, while at least some sparse tensors and vectors are still host/shared-memory objects.
### Expected Behavior
A one-rank complex CUDA/NCCL AFQMC run with an HDF5 `Factorized` Hamiltonian should either:
1. complete correctly on GPU, or
2. use an explicit supported fallback for this sparse host-backed path, rather than crashing or entering unsupported host/device mixed operations.
### Root Cause Hypothesis
In `SparseTensor.hpp`, the Factorized path appears to use host/shared CSR data structures such as `Spvn_view` and `SpvnT_view`, but `vHS`, `vbias`, and part of `energy` can receive CUDA device dense buffers.
The old code then constructs host-looking array refs or calls generic `ma::product` on a mix like:
- host/shared sparse matrix
- device dense matrix
- host vector or device output
That mix is not supported by the current CUDA matrix dispatch and can segfault or fall into the CUDA GEMV catch-all.
### Local Workaround Patch
The local workaround is deliberately conservative:
- Add `#include `.
- Under `ENABLE_CUDA || BUILD_AFQMC_HIP`, in `SparseTensor::vHS`:
- copy dense input `X` from device to a host scratch buffer,
- copy `v` to host scratch only when `c != 0`,
- run the existing host sparse multiply against `Spvn_view`,
- copy the local result segment back to the original output buffer.
- Do the same in `SparseTensor::vbias` for `G` and `SpvnT_view[k]`.
- In `SparseTensor::energy`:
- copy `Gc` to host scratch,
- compute into a host `E` scratch using the existing host path,
- copy `E` back to the original output buffer before returning.
This is not meant to be the fastest final solution. It is a correctness-first fallback for the host-backed sparse Factorized path. A proper device sparse implementation may be preferable if maintainers want this path to stay fully GPU-resident.
### Validation After Local Patch
With the local workaround patch and a rebuilt complex CUDA/NCCL AFQMC binary:
```text
Complex AFQMC CUDA/NCCL HDF5 Factorized/Cholesky, 1 MPI rank: PASS
Complex AFQMC CUDA/NCCL HDF5 THC, 1 MPI rank: PASS
Complex AFQMC CPU/PHDF5 HDF5 Factorized/Cholesky, 1 MPI rank: PASS
Real AFQMC CUDA/NCCL HDF5 Factorized, 1 MPI rank: PASS
Regular VMC CPU smoke: PASS
Regular VMC GPU smoke: PASS
```
The patched complex GPU Factorized run reaches:
```text
Energy of starting determinant:
- Total energy : (-10.3197575584,0)
- One-body energy : (-10.8097667596,0)
- Coulomb energy : (1.46758186973,0)
- Exchange energy : (-0.977572668585,0)
QMCPACK execution completed successfully
```
### Separate Upstream Limitation
This issue is separate from the existing multi-rank Factorized limitation:
```text
Error: Distributed Factorized hamiltonian not yet implemented.
```
That still occurs for a single complex Factorized input launched with multiple MPI ranks. The local fix above only addresses the one-rank CUDA/NCCL host/device SparseTensor failure.
### Question For Maintainers
Would a PR with the conservative host-scratch fallback in `SparseTensor::{energy,vHS,vbias}` be acceptable, or would you prefer a different implementation strategy, such as:
- adding a true device sparse Factorized path,
- explicitly forcing this path to CPU/host execution,
- or rejecting unsupported host/device combinations earlier with a clearer error?
I can prepare a PR with the patch and a small regression test if this direction is acceptable.
## Where To Submit
Submit this as a new issue at:
https://github.com/QMCPACK/qmcpack/issues
Repository confirmed from the local upstream remote:
```text
https://github.com/QMCPACK/qmcpack.git
```
Contributor guide
Research direction
Start with AFQMC/HamiltonianOperations/SparseTensor.hpp and trace SparseTensor::vHS, vbias, and energy through NOMSD::vMF and PropagatorFactory::buildAFQMCPropagator. Reproduce the one-rank complex CUDA/NCCL Factorized command, compare it with the listed CPU and THC smoke tests, and use the host/device mismatch as the failure check. Done means the Factorized run completes without mixed-memory failures and the chosen implementation has regression coverage.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- distributed-systems, hpc, performance
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100