lance-format / lance-format/lance

Refactor per-query ANN I/O metrics to the object-store level by reusing IOTrackingStore

Open
#7,224 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

chore
Dominant language
Rust
Stars
7.1k
Forks
852
Avg merge
3d 18h
Merged PRs (30d)
272

Description

Context

PR #7204 (closes #7201) made per-query index I/O observable on the ANNSubIndex and ANNIvfPartition operators in EXPLAIN ANALYZE. Because the ANN operators delegate reads to a cached, shared IVFIndex whose ScanScheduler is reused across queries, the established "own a fresh scheduler and read its cumulative stats" pattern did not apply. The PR introduced a lightweight per-query sink, IoStatsRecorder (a new trait in lance-core) backed by lance_io::scheduler::IoStats, attached at the FileScheduler accounting point.

In the review of that PR (https://github.com/lance-format/lance/pull/7204#pullrequestreview-4470060186), @wjones127 approved the approach as acceptable for now but asked for a future refactor that moves this tracking to the object-store level and reuses the existing IOTrackingStore / IOTracker rather than the scheduler-level hook.

Proposed direction

From the review:

Perhaps IOTrackingStore could hold a Vec<Arc<IOTracker>> and ObjectStore could have some method like:

impl ObjectStore {
    fn with_io_tracker(tracker: Arc<IOTracker>) -> Self { ... }
}

So you can always wrap that.

IOTracker already wraps an object store and accumulates IoStats (rust/lance-io/src/utils/tracking_store.rs). The idea is to let a caller attach a per-scope IOTracker to the object store an operator reads through, so per-query metrics fall out of the existing object-store tracking machinery instead of a parallel scheduler-level sink.

Acceptance criteria / open questions

  • Per-query bytes_read / iops / requests on ANNSubIndex and ANNIvfPartition stay exact and ~0 on a warm cache (parity with the tests added in #7204).
  • Metrics remain attributed per query, not process-wide cumulative, despite the shared/cached IVFIndex scheduler.
  • Evaluate whether IoStatsRecorder (lance-core) and the scheduler-level hook can be removed once the object-store path covers this, or whether both must coexist.
  • Confirm the object-store-level counts match what the FileScheduler accounting point reports today (post-coalescing physical I/O).

Follow-up to #7204. Optional / non-blocking refactor.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with rust/lance-io/src/utils/tracking_store.rs and the existing IOTracker/IOTrackingStore path, then inspect IoStatsRecorder in lance-core and the ANNSubIndex and ANNIvfPartition metrics from #7204. Run the per-query ANN I/O tests added in #7204 as a baseline. Done means object-store tracking preserves exact per-query bytes_read, iops, and requests, remains near zero on warm cache, and the scheduler hook is removed or its coexistence is justified.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.