Training techniques: natural gradients, hybrid optimizers, and more

Ideas for improving the training pipeline beyond what ptgp currently ships (compile_training_step with Adam, compile_scipy_objective with L-BFGS-B, minimize_staged_vfe).

• Natural gradients for SVGP — update q_mu/q_sqrt in natural parameter space. Typically 10-100x fewer iterations to the same ELBO. The param_groups infrastructure already supports separate learning rates; this needs a custom optimizer that computes the Fisher-metric inverse for the variational parameters.
• Adam warmup + L-BFGS-B polish — Adam for the first ~200 steps to get into the right basin, then L-BFGS-B for precise convergence. Handles mixed-scale optimization (hyperparams + inducing points) more robustly than L-BFGS-B alone.

What else?

What other training techniques or optimizer improvements would be useful?

Start by reading the existing entry points compile_training_step, compile_scipy_objective, and minimize_staged_vfe, along with the param_groups infrastructure. The issue proposes several optimizer directions but names no files, tests, or acceptance criteria, so the implementation scope and definition of done still need to be established.