deepmodeling / deepmodeling/deepmd-kit

docs: document features added since v3.1.3

Open
#5,917 0 comments 0 reactions 0 assignees View on GitHub
Docs
Dominant language
Python
Stars
2k
Forks
649
Avg merge
6d 18h
Merged PRs (30d)
15

Description

## Summary

While reviewing `feat(...)` commits added since the last stable release (`v3.1.3`), I found a number of user-facing features that are not documented in `doc/` (or only appear as schema entries in the auto-generated argument reference from `deepmd/utils/argcheck.py`, with no explanatory prose, examples, or guide-page coverage).

This issue tracks the documentation gaps so they can be closed. A checklist of items to add is at the bottom.

## Fully undocumented features

### 1. TF2 (TensorFlow 2) eager backend + training workflow — `dp --tf2`

- Commits: `73de44b1f` (eager Array backend, SavedModel export), `26ad0fb83` (training/freeze/compress workflow, multi-task, finetune, TensorBoard)
- Implementation: `deepmd/backend/tf2.py:32` (`name = "TensorFlow2"`), the `deepmd/tf2/` package, `deepmd/tf2/entrypoints/`
- Gap: Not listed in `doc/backend.md` (only TF v1 / PyTorch / JAX / Paddle / DP), no tab in `doc/train/training.md`, `doc/freeze/freeze.md`, or `doc/freeze/compress.md`. The `DP_JIT` env var is documented (`doc/env.md:51`), but the backend itself is not.

### 2. `pt_expt` (PyTorch-Exportable) backend — `dp --pt_expt`

- Commits: `9bd1f16b1`, `2cbef365d`, `3f91293b8`, `23622361d`, etc.
- Implementation: `deepmd/backend/pt_expt.py:32` (`name = "PyTorch-Exportable"`), `deepmd/pt_expt/`
- Gap: Referenced in `doc/model/dpa2.md`, `doc/model/dpa4.md`, `doc/model/train-se-atten.md` via the `--lower-kind graph` / `.pt2` graph-native route, but **never introduced as a backend** in `doc/backend.md`. Its distinct capabilities (`.pt2`/AOTInductor export, LMDB datasets, multi-task training, `dp finetune`/`change-bias`/`compress` for pt_expt) have no coherent documentation. `doc/freeze/freeze.md` only shows TF/PT/Paddle.

### 3. JAX training workflow — `dp --jax train`

- Commit: `20874166a` (`feat(jax): add training`)
- Gap: The doc change `1de6de9c9` only *deleted* the line "this backend … has no support for training" from `doc/backend.md`; it added no positive statement. `doc/train/training.md` has TF/PyTorch/Paddle tabs but **no JAX tab**, and `dp --jax train` is not documented anywhere.

### 4. `dp test` stress output

- Commit: `330fa7558` (`feat: dp test stress`)
- Implementation: `deepmd/entrypoints/test.py:540` (`.s.out` detail file), `:825`–`:835`, `:902`, `:976` (Stress MAE/RMSE in eV/ų)
- Gap: `doc/test/test.md` has no mention of stress output files or stress metrics; `grep stress doc/test/` returns nothing.

### 5. `DeepEval.serialize` / `dp show serialization-tree`

- Commit: `e19bd188b` (`feat(show): add serialization-tree via DeepEval.serialize`)
- Implementation: `deepmd/main.py:1049` (adds the `"serialization-tree"` choice), `deepmd/entrypoints/show.py:153`
- Gap: `doc/model/show-model-info.md` lists `model-branch`, `type-map`, `descriptor`, `fitting-net`, `size`, `observed-type` — but **not** `serialization-tree`. No prose anywhere.

### 6. `dp freeze --hessian` (JAX backend)

- Commit: `44131151f` (`feat(jax): freeze models with hessian output`)
- Implementation: `deepmd/main.py:354` (`parser_frz` `--hessian` flag, "Supported backend: JAX")
- Gap: `doc/freeze/freeze.md` does not document `--hessian`. Worse, `doc/model/train-energy-hessian.md:94` and `doc/model/overall.md:61` still state Hessian models **cannot be frozen** — now incorrect for JAX.

### 7. Vesin neighbor list + pluggable NeighborList strategy (Python/ASE inference)

- Commit: `23622361d` (`feat(pt_expt): pluggable NeighborList strategy + O(N) vesin neighbor list for Python/ASE inference`)
- Implementation: `deepmd/pt_expt/utils/vesin_neighbor_list.py`, `deepmd/pt_expt/infer/deep_eval.py:190`/`:214`/`:280` (`nlist_backend` argument: `"auto"`/`"vesin"`/`"native"`)
- Gap: `doc/inference/python.md:61` ("External neighbor list algorithm") is TF-only and documents only ASE `NewPrimitiveNeighborList`; `vesin` appears nowhere in `doc/`.

## Thinly documented features (schema-only / passing mention)

These are registered in `deepmd/utils/argcheck.py`, so they appear in the auto-generated argument reference, but they have no explanatory prose, examples, or guide-page coverage.

### 8. Descriptor compression for dpmodel & JAX backends — `dp --dp compress`, `dp --jax compress`

- Commit: `f143171cf` (`feat(dpmodel): add descriptor compression`)
- Implementation: `deepmd/dpmodel/entrypoints/compress.py`, `deepmd/dpmodel/entrypoints/compress_common.py`
- Gap: `doc/freeze/compress.md:4` states "Supported backends: TensorFlow, PyTorch" and only shows `dp compress` / `dp --pt compress`. The new dpmodel/JAX routes are not mentioned.

### 9. Charge-spin conditioning (training-side config) — `add_chg_spin_ebd`, `default_chg_spin`, `dim_chg_spin`

- Commits: `e3fda4537` (C++ runtime), `f39a08128` (dpa3 decouple from fparam), `6984e4c63` (paddle)
- Implementation: `deepmd/utils/argcheck.py:694` (`doc_add_chg_spin_ebd`), `:1041`, `:2097`
- Gap: The LAMMPS *inference* keyword `charge_spin` is documented (`doc/third-party/lammps-command.md:66`), and DPA4 mentions `add_chg_spin_ebd` only for graph-eligibility (`doc/model/dpa4.md:506,603`). The **training config** is undocumented in `doc/model/dpa3.md` and `doc/model/train-energy-spin.md` (both have zero mentions of `chg_spin`/`add_chg_spin`).

### 10. `stat_file` / `stat_file_mode` training config (TensorFlow)

- Commit: `2b3bb05a7` (`feat(tf): support training stat_file`)
- Implementation: `deepmd/utils/argcheck.py:5300` (`doc_stat_file`), `:5348` (Argument), `deepmd/tf/model/stat_file.py`
- Gap: No prose in `doc/train/`; `grep stat_file doc/` returns nothing.

### 11. `so3_readout` DPA4/SeZM option (`"none"`/`"glu"`/`"mlp"`)

- Commit: `4b6506db4` (`feat(dpa4): so3_readout across pt + dpmodel/pt_expt backends`)
- Implementation: `deepmd/utils/argcheck.py:626` (`doc_so3_readout`), `:999` (Argument)
- Gap: Not in `doc/model/dpa4.md` prose (the page documents many other DPA4 options).

### 12. EMA (Exponential Moving Average) shadow model — `enable_ema`, `ema_decay`, `ema_ckpt_keep`

- Commit: `57f870ff4` (`feat(pt): add ema shadow model`)
- Implementation: `deepmd/utils/argcheck.py:5264` (`doc_enable_ema`), `:5269` (`doc_ema_decay`)
- Gap: No prose in `doc/train/`. The only doc mention is a passing reference to `ema_ckpt_keep` inside the `ckpt_keep_ratio` description (`doc/train/training-advanced.md:107`).

### 13. Force L2-norm / force-MAE loss option — `force_l2_norm`

- Commit: `3f52fa95a` (`feat(dp, pt): add force l2 norm loss & mae loss`)
- Implementation: `deepmd/utils/argcheck.py:4326` ("use L2 norm of force vectors for loss calculation …")
- Gap: Not in `doc/model/train-energy.md` loss prose.

### 14. Three-value Huber delta — `huber_delta`

- Commit: `14c349b42` (`feat(loss): support three-value huber delta`)
- Implementation: `deepmd/utils/argcheck.py:4482` (`huber_delta` Argument)
- Gap: `grep huber doc/` returns nothing.

### 15. Element type-wise bias in property fitting — `distinguish_types`

- Commit: `7cd0d191b` (`feat(property): Support element type-wise bias in property fitting`)
- Implementation: `deepmd/utils/argcheck.py:2827` (`doc_distinguish_types`), `:2872`
- Gap: Not in `doc/model/train-fitting-property.md`.

### 16. LMDB dataset support for training

- Commits: `565f4beec` (pt/dpmodel dataloader), `54f42d90c` (pt_expt), `9d6381602` (max/filter mode), `a3f548d7b` (full validation lmdb)
- Implementation: `deepmd/pt_expt/utils/` (LMDB dataset), referenced from pt/dpmodel dataloaders
- Gap: Not in `doc/data/` or `doc/train/` as a training data format. Mentioned only in `doc/dpa_adapt/input_formats.md:105` (as a dpa-adapt conversion target).

## Orphaned / navigation-broken

### 17. `doc/model/train-fitting-population.md` is missing from the toctree

- `doc/model/train-fitting-population.md` exists and is well written, but it is **not listed in `doc/model/index.rst`** toctree (which has `train-fitting-tensor`, `train-fitting-dos`, `train-fitting-property` but omits `train-fitting-population`). `grep train-fitting-population doc/` finds no toctree reference, so the page is unreachable from navigation.

## Stale / now-inaccurate docs to fix

### 18. Hessian freeze claims are outdated

- `doc/model/train-energy-hessian.md:94` and `doc/model/overall.md:61` claim Hessian models cannot be frozen, which `dp freeze --hessian` (JAX) now contradicts (see item 6).

## Already documented (no action needed)

DPA4/SeZM model (`doc/model/dpa4.md`), NeighborGraph graph-native inference route (`doc/model/dpa2.md:94`, `doc/model/dpa4.md:499`), multi-rank LAMMPS inference, DPA-ADAPT toolkit (`doc/dpa_adapt/overview.md`), JAX-MD interface (`doc/third-party/jaxmd.md`), atomic charge population fitting (content good — only the toctree entry is missing, see item 17), property fitting, `shared_dict` in multi-task, `dp embed` embedding sub-CLI, plugin mechanism / entry-point loading, LAMMPS dynamic backend loading, `dE/dfparam` to LAMMPS via finite differences, PyTorch custom ops enabled by default, WSD LR scheduler, charge-spin LAMMPS inference keyword.

## Notes

- **Internal/optimization-only (probably should not be user-documented):** backend-independent trainer abstraction (`55d7e7991`, `deepmd/dpmodel/`); adaptive Array API neighbor search (`2a47cdc11` — internal dense/cell-list dispatcher); `feat(core): batch prod env mat over frames` (`5e80487fe`).
- **Experimental/partial:** JAX DPA4 descriptor (`c202c7b38`) is descriptor-only (no fitting/model/trainer), so `dpa4.md`'s "PyTorch only" statement is still accurate for end-to-end training; not a doc gap yet. The dipole-charge adapter commit `31221382c` extends an already-documented modifier (`doc/model/dplr.md`) to dpmodel/JAX/TF2 backends — the modifier itself is documented.
- **Schema vs. prose caveat:** the "thinly documented" items appear in the auto-generated argument reference built by sphinx-dargs, but have no explanatory prose, examples, or guide-page coverage.

## Documentation checklist

### Fully undocumented features

- [ ] Document the TF2 eager backend and `dp --tf2` training/freeze/compress workflow (add to `doc/backend.md`, `doc/train/training.md`, `doc/freeze/freeze.md`, `doc/freeze/compress.md`) — `73de44b1f`, `26ad0fb83`
- [ ] Introduce the `pt_expt` (PyTorch-Exportable) backend in `doc/backend.md` and document its distinct capabilities (`.pt2`/AOTInductor export, LMDB datasets, multi-task, `dp finetune`/`change-bias`/`compress`) — `9bd1f16b1` et al.
- [ ] Add a JAX tab to `doc/train/training.md` and document `dp --jax train` — `20874166a`
- [ ] Document `dp test` stress output (`.s.out` file, Stress MAE/RMSE in eV/ų) in `doc/test/test.md` — `330fa7558`
- [ ] Add `serialization-tree` to the list of `dp show` choices in `doc/model/show-model-info.md` — `e19bd188b`
- [ ] Document `dp freeze --hessian` (JAX) in `doc/freeze/freeze.md` — `44131151f`
- [ ] Document the vesin neighbor list and `nlist_backend` (`auto`/`vesin`/`native`) pluggable strategy in `doc/inference/python.md` — `23622361d`

### Thinly documented features (add prose / examples)

- [ ] Extend `doc/freeze/compress.md` to cover dpmodel/JAX compression (`dp --dp compress`, `dp --jax compress`) — `f143171cf`
- [ ] Document the charge-spin training config (`add_chg_spin_ebd`, `default_chg_spin`, `dim_chg_spin`) in `doc/model/dpa3.md` and `doc/model/train-energy-spin.md` — `e3fda4537`, `f39a08128`
- [ ] Document `stat_file` / `stat_file_mode` (TF training) in `doc/train/` — `2b3bb05a7`
- [ ] Document the `so3_readout` DPA4/SeZM option in `doc/model/dpa4.md` — `4b6506db4`
- [ ] Document EMA (`enable_ema`, `ema_decay`, `ema_ckpt_keep`) in `doc/train/training-advanced.md` — `57f870ff4`
- [ ] Document `force_l2_norm` in `doc/model/train-energy.md` loss section — `3f52fa95a`
- [ ] Document `huber_delta` in the loss section — `14c349b42`
- [ ] Document `distinguish_types` (element type-wise bias) in `doc/model/train-fitting-property.md` — `7cd0d191b`
- [ ] Document LMDB dataset support in `doc/data/` and `doc/train/` — `565f4beec`, `54f42d90c`

### Orphaned / navigation-broken

- [ ] Add `train-fitting-population` to the `doc/model/index.rst` toctree

### Stale / now-inaccurate docs to fix

- [ ] Update `doc/model/train-energy-hessian.md:94` and `doc/model/overall.md:61` — Hessian models can now be frozen via `dp freeze --hessian` (JAX)

---

Coding agent: opencode
opencode version: 1.18.8
Model: ustc/glm-5.2
Reasoning effort: max

Contributor guide

Open the contributing guide

Research direction

Start with one checklist item, reading its named implementation entry point alongside the target documentation file, such as deepmd/backend/tf2.py and doc/backend.md. Update the relevant guide pages, examples, and stale statements, then verify that doc/model/index.rst exposes the population-fitting page and that each selected feature is described where the checklist specifies.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch, tensorflow
Domain
documentation, machine-learning
Issue type
Documentation
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.