lance-format / lance-format/lance
Refactor per-query ANN I/O metrics to the object-store level by reusing IOTrackingStore
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
Context
PR #7204 (closes #7201) made per-query index I/O observable on the ANNSubIndex and ANNIvfPartition operators in EXPLAIN ANALYZE. Because the ANN operators delegate reads to a cached, shared IVFIndex whose ScanScheduler is reused across queries, the established "own a fresh scheduler and read its cumulative stats" pattern did not apply. The PR introduced a lightweight per-query sink, IoStatsRecorder (a new trait in lance-core) backed by lance_io::scheduler::IoStats, attached at the FileScheduler accounting point.
In the review of that PR (https://github.com/lance-format/lance/pull/7204#pullrequestreview-4470060186), @wjones127 approved the approach as acceptable for now but asked for a future refactor that moves this tracking to the object-store level and reuses the existing IOTrackingStore / IOTracker rather than the scheduler-level hook.
Proposed direction
From the review:
Perhaps
IOTrackingStorecould hold aVec<Arc<IOTracker>>andObjectStorecould have some method like:impl ObjectStore { fn with_io_tracker(tracker: Arc<IOTracker>) -> Self { ... } }So you can always wrap that.
IOTracker already wraps an object store and accumulates IoStats (rust/lance-io/src/utils/tracking_store.rs). The idea is to let a caller attach a per-scope IOTracker to the object store an operator reads through, so per-query metrics fall out of the existing object-store tracking machinery instead of a parallel scheduler-level sink.
Acceptance criteria / open questions
- Per-query
bytes_read/iops/requestsonANNSubIndexandANNIvfPartitionstay exact and ~0 on a warm cache (parity with the tests added in #7204). - Metrics remain attributed per query, not process-wide cumulative, despite the shared/cached
IVFIndexscheduler. - Evaluate whether
IoStatsRecorder(lance-core) and the scheduler-level hook can be removed once the object-store path covers this, or whether both must coexist. - Confirm the object-store-level counts match what the
FileScheduleraccounting point reports today (post-coalescing physical I/O).
Follow-up to #7204. Optional / non-blocking refactor.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with rust/lance-io/src/utils/tracking_store.rs and the existing IOTracker/IOTrackingStore path, then inspect IoStatsRecorder in lance-core and the ANNSubIndex and ANNIvfPartition metrics from #7204. Run the per-query ANN I/O tests added in #7204 as a baseline. Done means object-store tracking preserves exact per-query bytes_read, iops, and requests, remains near zero on warm cache, and the scheduler hook is removed or its coexistence is justified.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- databases
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100