[challenge]: Improving VMC from an RL perspective, benchmarked on the 1/8-doped Hubbard model

Released by

Huanhai ZHOU, the Hong Kong University of Science and Technology (Guangzhou)

Contact email

albus.zhouhh@gmail.com

Method

Variational Monte Carlo / Neural Quantum States

Challenge issue

The issue

Variational Monte Carlo is, structurally, reinforcement learning. The Born distribution $\pi_\theta(x) = |\psi_\theta(x)|^2/Z$ is the policy of a one-step task with an exponentially large action space — one configuration $x$ per action; minus the local energy $E_{\mathrm{loc}}(x) = \langle x|H|\psi_\theta\rangle/\langle x|\psi_\theta\rangle$ plays the role of the reward; the gradient of the variational energy $E = \mathbb{E}_{x\sim\pi_\theta}[E_{\mathrm{loc}}(x)]$,

$$\nabla_\theta E = 2,\mathrm{Re};\mathbb{E}{x\sim\pi\theta}\Big[\big(E_{\mathrm{loc}}(x) - \mathbb{E}[E_{\mathrm{loc}}]\big),\nabla_\theta\log\psi_\theta^*(x)\Big]$$

is exactly REINFORCE with a mean baseline, and stochastic reconfiguration (SR) [8] is natural-gradient descent [9] — the same idea that underlies trust-region policy optimization (TRPO) in RL [6]. When the ansatz is autoregressive, sampling becomes a sequence of site-level decisions with known deterministic transitions and reward only at the end — precisely the setting of modern sequence-level RL. A very recent demonstration [5] adapts the ratio clipping of proximal policy optimization (PPO) [7] to this setting and obtains a first-order optimizer with no curvature matrix and no linear solve, which reuses samples across updates and trains ansätze three orders of magnitude larger than SR-family methods can handle.

Seen from this angle, VMC as commonly practiced occupies one narrow corner of the RL design space, and that corner has known pathologies:

• One model is both the variational state and its own sampler. Training is strictly on-policy: configurations are drawn from $|\psi_\theta|^2$ itself. No mechanism balances exploration against exploitation. When competing low-energy states are separated by many local moves — stripe states of different wavelength, for example — the sampler stops visiting the modes the current state underweights, and the estimated gradient no longer carries the information needed to escape: mode collapse with biased gradients.
• The natural-gradient step is estimated, not evaluated. An SR update is a step of imaginary-time evolution restricted to the ansatz. Tensor-network methods evaluate the expectation values defining this step by exact contraction; VMC estimates the same quantities — metric and force — from finite on-policy samples and pushes them through a regularized linear solve. Statistical noise, regularization bias, and step-size heuristics are consequences of the estimator, not of the variational principle.
• Born sampling is not variance-optimal, and raw $E_{\mathrm{loc}}$ is heavy-tailed near fermionic nodes, where $|\psi_\theta(x)|$ is small.

Possibilities (illustrations, not prescriptions)

Nothing below is required; the list only illustrates that the perspective generates concrete, non-exotic proposals. One structural advantage runs through all of them: VMC can evaluate density ratios $|\psi_{\mathrm{new}}(x)/\psi_{\mathrm{old}}(x)|^2$ exactly, so off-policy machinery can be imported without the approximations RL is normally forced to make.

• Off-policy estimation [12] — replay of previously sampled configurations, or explicit behavior distributions (the tempered $|\psi|^{2\beta}$ with $\beta < 1$, which flattens the distribution; mixtures seeded with competing stripe patterns), corrected by exact importance ratios; targets mode collapse and sampling cost at once.
• Reward processing [11] — RL never feeds raw rewards to the gradient: normalization and clipping are standard, carefully studied components, including the bias they introduce. VMC codes clip heavy-tailed $E_{\mathrm{loc}}$ as undocumented folklore; the RL literature offers principled versions.
• Credit assignment for autoregressive ansätze [10] — with terminal-only reward on known deterministic dynamics, prefix value functions $V(x_1\ldots x_k)$ (conditional energies given a partial configuration) enable per-site advantages and lower-variance gradients.
• Learned proposal kernels [13] — the Metropolis proposal is itself a policy with a measurable objective (mixing speed); training it, for a given Hamiltonian or across Hamiltonians, attacks mode collapse at the sampler rather than at the state.

Benchmark

The benchmark is the two-dimensional Hubbard model

$$H = -t\sum_{\langle ij\rangle\sigma}\big(c^\dagger_{i\sigma}c_{j\sigma} + \mathrm{h.c.}\big) - t'\sum_{\langle\langle ij\rangle\rangle\sigma}\big(c^\dagger_{i\sigma}c_{j\sigma} + \mathrm{h.c.}\big) + U\sum_i n_{i\uparrow}n_{i\downarrow},$$

with nearest-neighbor hopping $t = 1$, on-site repulsion $U = 8$, next-nearest-neighbor hopping $t' \in {0, -0.2}$, at hole doping $\delta = 1/8$ (electron density $n = 0.875$). In this regime stripe order is the established multi-method consensus [2], and the strongest published variational results are those of Ref. [1] (transformer-based neural quantum states, NQS) together with the density-matrix renormalization group (DMRG) and projected entangled-pair state (PEPS) results quoted therein. DMRG gives the lowest energies on narrow systems ($L_y \le 6$), NQS on wider ones — no method currently dominates at all sizes.

Reference geometries: every $\delta = 1/8$ geometry with energies reported in Ref. [1] — at $t' = 0$, the $16\times L_y$ open-boundary series ($L_y = 4, 6, 8, 10, 12, 16$; DMRG comparison at $L_y \le 8$, PEPS at all widths) and the periodic $8\times 8$, $12\times 12$, $16\times 16$ lattices; at $t' = -0.2$, the periodic $L_x \times 8$ series ($L_x = 8$ to $32$) and the $16\times 10$ and $16\times 12$ lattices.

Comparison rule: raw variational energies with statistical error bars. Extrapolated estimates (bond dimension $\to \infty$, etc.) are not variational states and are excluded from the bar. Where the best reported number is already quasi-exact, "not worse" means matching within errors — the variational principle forbids more.

Certification

• Validation. $8\times 8$, periodic boundaries, $U = 8$, half filling: reproduce the quasi-exact auxiliary-field QMC benchmarks [3,4] within error bars. A correctness gate for the full pipeline — half filling is free of the fermion sign problem and well charted, so no algorithmic credit is earned here.
• Minimal target. $16\times 4$ lattice, open boundaries, $\delta = 1/8$, $t' = 0$: outperform a control run that uses the same ansatz and the same sample budget optimized with standard SR (or a scalable SR variant) — lower converged energy at matched budget, or matched energy at reduced cost. This ablation is what separates an algorithmic advance from gains bought by a larger ansatz or more compute.
• Full target. At every reference geometry above, variational energy not worse than the best variational result reported in Ref. [1] (NQS, or the DMRG/PEPS quoted therein) within statistical errors.

Conditions for a certified submission: one ansatz family across all geometries (size-scaling of the same architecture is allowed), and reported compute (sample counts, hardware, wall time). Recommended: report hole-density profiles — the stripe wavelength is the natural mode-collapse diagnostic in this regime, visible where the energy alone is silent.

Why this may lead to research output

• Better variational energies on a problem that matters. The doped Hubbard model is central to quantum many-body physics and tied to the understanding of cuprate high-$T_c$ superconductivity [2]. Any certified improvement is a better variational algorithm on a problem where the strongest methods still disagree; the full target would be state of the art.
• The perspective itself is productive. Viewing VMC as RL turns algorithm design into separately investigable axes — exploration, exploitation, estimation — instead of closing the question with "SR is imaginary-time evolution and therefore already optimal."

References

1. Y. Gu et al., Solving the Hubbard model with Neural Quantum States, arXiv:2507.02644.
2. B.-X. Zheng et al., Stripe order in the underdoped Hubbard model, Science 358, 1155 (2017).
3. M. Qin, H. Shi, and S. Zhang, Benchmark study of the two-dimensional Hubbard model with auxiliary-field quantum Monte Carlo, Phys. Rev. B 94, 085103 (2016).
4. J. P. F. LeBlanc et al. (Simons Collaboration), Solutions of the two-dimensional Hubbard model: benchmarks and results from a wide range of numerical algorithms, Phys. Rev. X 5, 041041 (2015).
5. J. A. Duque et al., One More Time: Revisiting Neural Quantum States from a Reinforcement Learning Perspective, arXiv:2607.02292.
6. J. Schulman et al., Trust Region Policy Optimization, arXiv:1502.05477.
7. J. Schulman et al., Proximal Policy Optimization Algorithms, arXiv:1707.06347.
8. S. Sorella, Green function Monte Carlo with stochastic reconfiguration, Phys. Rev. Lett. 80, 4558 (1998).
9. S.-i. Amari, Natural gradient works efficiently in learning, Neural Computation 10, 251 (1998).
10. J. Schulman et al., High-Dimensional Continuous Control Using Generalized Advantage Estimation, arXiv:1506.02438.
11. Z. Shao et al., DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models (GRPO), arXiv:2402.03300.
12. T. Degris, M. White, and R. S. Sutton, Off-Policy Actor-Critic, arXiv:1205.4839.
13. J. Liu, Y. Qi, Z. Y. Meng, and L. Fu, Self-learning Monte Carlo method, Phys. Rev. B 95, 041101(R) (2017).

The issue names no repository files, tests, or implementation entry points. Start by locating the existing Python VMC/NQS and SR benchmark workflow, then establish the half-filled 8×8 validation before comparing methods on the 16×4 doped Hubbard target. Done requires certified energy comparisons, statistical errors, matched budgets, one ansatz family, and reported compute.