Core implementations to add: practical GP advances

A list of implementations that would make ptgp more useful as a general-purpose GP toolkit. The goal is broadly useful GP advances, not things hyper-targeted to specific problems. ptgp should also work well as a substrate for people building specialized models on top, but the core should cover what practitioners should reach for more often.

Objectives

• LOO-CV (Leave-One-Out Cross-Validation) — analytic for exact GPs with Gaussian noise. Penalizes overfitting more aggressively than MLL. Useful for kernel model selection without a validation set.
• Predictive log likelihood (PLL) for SVGP — Jankowiak et al. (2020). Uses log E[p(y|f)] instead of E[log p(y|f)]. Tends to produce better-calibrated predictive variances than the standard ELBO.

Sparse / structured approximations

• Spherical harmonic features — Dutordoir et al. (2020). Inter-domain inducing variables on the unit hypersphere. Diagonal Kuu, so solves are elementwise division. Multi-dimensional counterpart to VFF.
• Iterative solvers (CG, Lanczos) — the GPyTorch approach: use conjugate gradients for solves and stochastic trace estimators for log determinants instead of Cholesky. Scales to much larger N than exact methods. How this should work in ptgp is an open question: should PyTensor rewrites automatically choose between Cholesky and CG based on problem size, or should this be a user-facing choice? See #9 for related discussion on rewrites.

What else?

What other broadly useful GP methods or features are missing?

Start by reviewing the proposed objectives in this issue and the related rewrite discussion in #9. No implementation files or tests are named; the work is not ready to begin until one method is selected, scoped, and given concrete completion criteria.