QMCPACK / QMCPACK/qmcpack

QMCPACK Issue Draft: Complex CUDA/NCCL AFQMC Factorized SparseTensor Crash

Open
#5,999 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
403
Forks
154
Avg merge
1d 13h
Merged PRs (30d)
90

Description

## Issue Body

### Summary

I hit a reproducible failure in QMCPACK 4.3.0 when running a complex-valued AFQMC calculation with an HDF5 `Factorized` Hamiltonian on the CUDA/NCCL build.

The same small complex AFQMC input runs with the complex CPU/PHDF5 build. The complex GPU/NCCL build also runs an HDF5 THC Hamiltonian input. The failing path appears specific to the complex CUDA/HIP `SparseTensor` / Factorized Hamiltonian path.

I have a local workaround patch that makes the one-rank complex GPU/NCCL Factorized/Cholesky smoke test complete successfully. I am opening this issue first to confirm whether the maintainers prefer this conservative host-scratch fallback, or a proper device sparse implementation.

### Environment

- QMCPACK: 4.3.0
- Upstream commit tested: `bb7eede051f98ec03296664b304982e655f960c4`
- Build mode:
- `QMC_COMPLEX=ON`
- `BUILD_AFQMC=ON`
- `ENABLE_PHDF5=ON`
- `ENABLE_CUDA=ON`
- `BUILD_AFQMC_WITH_NCCL=ON`
- GPU: NVIDIA A100-SXM4-80GB
- CUDA: 12.9
- NCCL: 2.25
- MPI: OpenMPI via Conda environment

### Minimal Run Shape

Input shape:

```xml


26
4
4


hdf5
../choldump.h5


ascii
../wfn.dat


closed


0.01
1
1
2

```

Command:

```bash
CUDA_VISIBLE_DEVICES=0 OMP_NUM_THREADS=1 \
mpiexec -n 1 --bind-to none \
/path/to/qmcpack qmc_short_chol.in.xml --gpu
```

### Observed Behavior

Before the local patch, the run failed in the complex CUDA/NCCL build during AFQMC propagator initialization / mean-field subtraction.

The crash mapped into the `SparseTensor` Factorized path, including:

- `AFQMC/HamiltonianOperations/SparseTensor.hpp`
- `SparseTensor::vbias`
- `SparseTensor::vHS`
- `NOMSD::vMF`
- `PropagatorFactory::buildAFQMCPropagator`

After patching only `vHS` and `vbias`, the code progressed further but then failed in the energy path with a host/device GEMV mismatch:

```text
types: (gemv) T: std::complex
ptrA: device::device_pointer>
ptrB: const std::complex*
ptrC: device::device_pointer>
Error: Calling qmc_cuda::gemv catch all.
```

This suggests that the Factorized `SparseTensor` path is passing CUDA device dense arrays into host sparse/dense operations, while at least some sparse tensors and vectors are still host/shared-memory objects.

### Expected Behavior

A one-rank complex CUDA/NCCL AFQMC run with an HDF5 `Factorized` Hamiltonian should either:

1. complete correctly on GPU, or
2. use an explicit supported fallback for this sparse host-backed path, rather than crashing or entering unsupported host/device mixed operations.

### Root Cause Hypothesis

In `SparseTensor.hpp`, the Factorized path appears to use host/shared CSR data structures such as `Spvn_view` and `SpvnT_view`, but `vHS`, `vbias`, and part of `energy` can receive CUDA device dense buffers.

The old code then constructs host-looking array refs or calls generic `ma::product` on a mix like:

- host/shared sparse matrix
- device dense matrix
- host vector or device output

That mix is not supported by the current CUDA matrix dispatch and can segfault or fall into the CUDA GEMV catch-all.

### Local Workaround Patch

The local workaround is deliberately conservative:

- Add `#include `.
- Under `ENABLE_CUDA || BUILD_AFQMC_HIP`, in `SparseTensor::vHS`:
- copy dense input `X` from device to a host scratch buffer,
- copy `v` to host scratch only when `c != 0`,
- run the existing host sparse multiply against `Spvn_view`,
- copy the local result segment back to the original output buffer.
- Do the same in `SparseTensor::vbias` for `G` and `SpvnT_view[k]`.
- In `SparseTensor::energy`:
- copy `Gc` to host scratch,
- compute into a host `E` scratch using the existing host path,
- copy `E` back to the original output buffer before returning.

This is not meant to be the fastest final solution. It is a correctness-first fallback for the host-backed sparse Factorized path. A proper device sparse implementation may be preferable if maintainers want this path to stay fully GPU-resident.

### Validation After Local Patch

With the local workaround patch and a rebuilt complex CUDA/NCCL AFQMC binary:

```text
Complex AFQMC CUDA/NCCL HDF5 Factorized/Cholesky, 1 MPI rank: PASS
Complex AFQMC CUDA/NCCL HDF5 THC, 1 MPI rank: PASS
Complex AFQMC CPU/PHDF5 HDF5 Factorized/Cholesky, 1 MPI rank: PASS
Real AFQMC CUDA/NCCL HDF5 Factorized, 1 MPI rank: PASS
Regular VMC CPU smoke: PASS
Regular VMC GPU smoke: PASS
```

The patched complex GPU Factorized run reaches:

```text
Energy of starting determinant:
- Total energy : (-10.3197575584,0)
- One-body energy : (-10.8097667596,0)
- Coulomb energy : (1.46758186973,0)
- Exchange energy : (-0.977572668585,0)

QMCPACK execution completed successfully
```

### Separate Upstream Limitation

This issue is separate from the existing multi-rank Factorized limitation:

```text
Error: Distributed Factorized hamiltonian not yet implemented.
```

That still occurs for a single complex Factorized input launched with multiple MPI ranks. The local fix above only addresses the one-rank CUDA/NCCL host/device SparseTensor failure.

### Question For Maintainers

Would a PR with the conservative host-scratch fallback in `SparseTensor::{energy,vHS,vbias}` be acceptable, or would you prefer a different implementation strategy, such as:

- adding a true device sparse Factorized path,
- explicitly forcing this path to CPU/host execution,
- or rejecting unsupported host/device combinations earlier with a clearer error?

I can prepare a PR with the patch and a small regression test if this direction is acceptable.

## Where To Submit

Submit this as a new issue at:

https://github.com/QMCPACK/qmcpack/issues

Repository confirmed from the local upstream remote:

```text
https://github.com/QMCPACK/qmcpack.git
```

Contributor guide

Open the contributing guide

Research direction

Start with AFQMC/HamiltonianOperations/SparseTensor.hpp and trace SparseTensor::vHS, vbias, and energy through NOMSD::vMF and PropagatorFactory::buildAFQMCPropagator. Reproduce the one-rank complex CUDA/NCCL Factorized command, compare it with the listed CPU and THC smoke tests, and use the host/device mismatch as the failure check. Done means the Factorized run completes without mixed-memory failures and the chosen implementation has regression coverage.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
distributed-systems, hpc, performance
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.