Tracking: Efficient Lazy/Chronic Computations in SortingAnalyzer

Overview

This issue tracks a series of improvements aimed at making SortingAnalyzer computations more efficient, particularly for long-term / chronic recordings and large-scale pipelines. The overarching goal is to reduce redundant I/O, enable incremental computation, and allow results to be written directly to efficient storage backends (e.g. Zarr) without unnecessary in-memory buffering.

Motivation

Current SortingAnalyzer workflows load and compute extensions eagerly, which becomes a bottleneck when:

• Working with chronic/long-term recordings that accumulate large amounts of waveform data
• Re-running partial computations (e.g. only a subset of units or a new time range changed)
• Saving results to Zarr-based pipelines where intermediate in-memory copies are wasteful

Efforts

1. Lazy Loading

PR: #4623

Support for lazily loading SortingAnalyzer extensions — extension data is not loaded into memory until explicitly accessed. This reduces startup time and memory footprint when only a subset of extensions are needed.

2. Lazy Compute

PR: #4713

Distinct from lazy loading: allows extension computations to be deferred and triggered on demand rather than eagerly computed when compute() is called. This enables incremental / on-demand pipelines without full recomputation.

3. Gather to Zarr

PR: #4703

Enables writing extension results directly to Zarr arrays during computation, bypassing in-memory accumulation. Critical for chronic recording workflows where the full result set does not fit in RAM.

4. Extract Waveforms to Zarr

PR: #4708

Extends the gather-to-Zarr backend to waveform extraction specifically, allowing ComputeWaveforms to stream extracted snippets directly into Zarr storage rather than accumulating them in memory first.

5. Optimize QC Computation

PR: #4715

6. Reduce UnitRefine function RAM (run from metrics)

PR: #4716

Quality metric computations are optimized to take advantage of the lazy/streaming infrastructure above, reducing redundant passes over waveform data and improving throughput for large unit counts.

Checklist

• #4623 — Lazy loading
• #4713 — Lazy compute
• #4703 — Gather to Zarr
• #4708 — Extract waveforms to Zarr
• #4715 — Optimize QC computation
• #4716 — Reduce UnitRefine function RAM (run from metrics)

Notes

• These PRs are interdependent: lazy compute and gather-to-Zarr both build on the lazy loading infrastructure.
• Testing should cover chronic recording use cases (many segments, large unit counts) to validate memory and performance gains end-to-end.

Start by reviewing PRs #4623, #4713, #4703, #4708, #4715, and #4716, then trace the SortingAnalyzer computation workflow they modify. Done means the tracked lazy, streaming, and memory-reduction improvements work together for chronic recordings and are covered by end-to-end tests measuring large-scale memory and performance behavior.