DPA1-L0-M PT-EXPT models fail in LAMMPS/Kokkos water MD

# DPA1-L0-M PT-EXPT models fail in LAMMPS/Kokkos water MD

## Summary

We observe two related failure modes when running a 192-atom, 64-water periodic system with LAMMPS + Kokkos + `deepmd/kk`:

1. The released pretrained ordinary DPA1-L0-M PT-EXPT model can complete a short 100-step smoke test, but a longer NVE trajectory becomes numerically unstable and then aborts during Kokkos neighbor-list rebuilding.
2. Our own 2M-step fine-tuned H/O DPA1-L0-M model fails before the first useful MD step in both ordinary and sampled-compressed PT-EXPT forms, for both NVE and NVT.

The minimal public reproduction repository contains the exact binary models, wrapped structure, portable LAMMPS inputs, variableized SLURM scripts, sanitized output excerpts, and hashes:

https://github.com/Zhang-pchao/deepmd-kokkos-pt2-reproduction

The water data source is the public Zenodo dataset:

https://zenodo.org/records/14780363

## Environment

- GPU: NVIDIA GeForce RTX 3090, 24 GiB
- NVIDIA driver: 550.90.07
- CUDA runtime: 12.6
- DeePMD-kit build: CUDA + LAMMPS + Kokkos
- LAMMPS: `4 Jul 2026`
- Kokkos: `5.1.99`

## Models

| Model | Artifact | SHA256 |
|---|---|---|
| Released pretrained ordinary PT-EXPT | `models/pretrained_ordinary.pt2` | `926cacf1d4460f3d90a1237e1c9f8bfdd89620a809f61ef37c6d003776705785` |
| Fine-tuned ordinary PT-EXPT | `models/finetuned_ordinary.pt2` | `32f789b4429ab3a9486d7e77cac749fe55d1b04be45b9930374f751f51f166bf` |
| Fine-tuned sampled-compressed PT-EXPT | `models/finetuned_compressed_sampled5.pt2` | `4832213dad3bb0e5358a0715001a42352938813dcbf80891800df97b133a7dc0` |

The pretrained artifact is the ordinary PT-EXPT conversion of the released pretrained checkpoint. The fine-tuned ordinary artifact is from a 2M-step DPA1-L0-M H/O fine-tuning run. Full-data compression exceeded the available 24 GiB GPU memory; the compressed artifact therefore used five evenly spaced frames from each of 21 training systems (105 frames total).

## System and input

- 192 atoms = 64 H2O molecules
- 2 atom types: H and O
- Periodic orthogonal box: 12.42 x 12.42 x 12.42 Angstrom
- LAMMPS mapping: `pair_coeff * * H O`
- `neighbor 1.0 bin`
- `neigh_modify every 1 delay 0 check yes`
- Timestep: 0.5 fs
- Initial velocities: 300 K
- NVE: 20,000 steps (10 ps)
- NVT: 300 K, Nose-Hoover damping 0.1 ps, 20,000 steps (10 ps)
- LAMMPS trajectory and XYZ output requested every 100 steps

The repository provides:

- `data/water64_wrapped.data`
- `inputs/in.nve.lammps`
- `inputs/in.nvt.lammps`
- `scripts/slurm_nve.sh`
- `scripts/slurm_nvt.sh`
- `outputs/` with sanitized logs for all tested variants

The SLURM scripts use `MODEL_PATH`, `DATA_PATH`, and `OUTPUT_PREFIX` variables. No private filesystem paths are embedded.

## Observed behavior

| Model and ensemble | Result |
|---|---|
| Released pretrained ordinary, NVE | 100-step smoke test completes; longer NVE heats rapidly and aborts around step 1100 after trajectory blow-up |
| Fine-tuned ordinary, NVE | Only step 0 is printed; no integration step completes |
| Fine-tuned ordinary, NVT | Only step 0 is printed; no integration step completes |
| Fine-tuned sampled-compressed, NVE | Only step 0 is printed; no integration step completes |
| Fine-tuned sampled-compressed, NVT | Only step 0 is printed; no integration step completes |

For the fine-tuned ordinary and sampled-compressed models, the same error appears during the first dynamic neighbor-list construction:

```text
cudaDeviceSynchronize() error( cudaErrorIllegalAddress): an illegal memory access was encountered
```

The relevant backtrace reaches:

```text
Kokkos::Impl::ExecSpaceManager::static_fence
LAMMPS_NS::NBinKokkos::bin_atoms
LAMMPS_NS::NeighborKokkos::build_kokkos
LAMMPS_NS::VerletKokkos::run
```

The fine-tuned jobs terminate with exit code 6. They produce only the initial step-0 record and no dynamic trajectory frame.

For the released pretrained ordinary model, the longer NVE run shows approximately:

```text
step 1000: temperature 4320.6462 K; total energy -956.17823 eV
step 1100: temperature 1018411.9 K; total energy 24590.473 eV
```

followed by the same `cudaErrorIllegalAddress` during Kokkos neighbor rebuilding.

## Questions

1. Is the Kokkos neighbor-list abort expected after model-driven trajectory divergence or pathological coordinates?
2. Should `deepmd/kk` or the Kokkos neighbor-list path detect this condition and report a clearer numerical/model error instead of an illegal memory access?
3. Why does the fine-tuned PT-EXPT model fail during the first dynamic neighbor-list construction in both ordinary and compressed forms?
4. Are there compatibility requirements for ordinary and compressed PT-EXPT artifacts on this LAMMPS/Kokkos/CUDA combination?
5. Is there a recommended CPU or non-Kokkos diagnostic that separates model extrapolation/physical instability from a Kokkos neighbor-list failure?

## Caveat

The released DPA1-MatPES model is trained and validated on the MatPES chemical space, and pure water may be outside its validated domain. The fine-tuned model is our own private training artifact, now published in the reproduction repository for this issue. This report does not claim physical accuracy for water; it reports the reproducible deployment failure and the opaque CUDA error.