Out-of-core minibatch ADVI on a financial tick stream

Notebook proposal

Title: Out-of-core minibatch ADVI on a financial tick stream

Why should this notebook be added to pymc-examples?

Once a dataset outgrows memory, PyMC users currently have no worked example to follow: every VI and minibatch example assumes the data sits in RAM. The DataLoader merged in pymc-devs/pymc-extras#698 removes that limit, and this notebook is its worked example, end to end on realistic financial tick data (fully self-contained: the data is generated inside the notebook, no downloads).

It demonstrates the parts that go wrong silently if skipped, which is what makes it a tutorial rather than an API tour: why the likelihood must factor over rows for minibatch rescaling to be valid (and why the classic stochastic-volatility model therefore cannot be minibatched — this notebook is the complement to that example, not a replacement); a disk-level shuffle for strongly ordered data, since a bounded buffer only block-shuffles; the total_size contract and what the mismatch looks like; a hurdle likelihood for the 30% of ticks with exactly zero price move; and loss-based stopping on a noisy streamed ELBO, including how a naive per-step rule false-stops.

Suggested categories:

• Level: intermediate
• Diataxis type: How-to

Related notebooks

• variational_api_quickstart — this builds directly on it, adding the out-of-core data path.
• stochastic_volatility — cited as the contrast case: a latent-path likelihood that does not factor over rows and so cannot be minibatched; this notebook explains why and shows a model class that can.
• GLM hierarchical / minibatch examples — same total_size machinery, in-memory; this notebook moves it out of core.
• Supersedes the draft in pymc-examples#888, which was written against the pre-merge DataLoader API; that PR will be closed pointing here so there is one example, not two.

References

• pymc-devs/pymc-extras#698 (the merged DataLoader), #733 (loss-based convergence callback, in review).
• Kucukelbir et al. (2015), Automatic Differentiation Variational Inference — the total_size minibatch rescaling.
• Page (1954), Continuous inspection schemes — the change-detection framing used in the stopping section.

Note on dependencies: the DataLoader is on pymc-extras main but not yet in a released version (latest is v0.14.0, 2026-07-28); the notebook targets the next release.

Start by reading the variational_api_quickstart, stochastic_volatility, and related GLM minibatch notebooks, then review the merged DataLoader in pymc-extras#698 and the loss-based callback in #733. The notebook is done when it self-generates financial tick data and demonstrates valid row-factorized minibatching, disk-level shuffling, total_size behavior, the hurdle likelihood, and robust streamed-ELBO stopping against the next dependency release.