lance-format / lance-format/lance
Reuse scan schedulers for additional base paths instead of creating one per fragment open
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
FileFragment::open_reader_impl creates a new ScanScheduler for every fragment open when the data file lives in an additional base path (rust/lance/src/dataset/fragment.rs, the data_file.base_id branch checks neither read_config.scan_scheduler nor any per-base cache).
Each ScanScheduler::new spawns an io-loop task and an independent IoQueue with max_bandwidth = 32 MiB × io_parallelism (cloud default 64 → ~2 GiB backpressure budget and 64 IOPS per scheduler). A scan with N shallow-clone fragments in flight therefore runs N io-loops with N independent quotas: the shared scheduler, the user-configured io_buffer_size, and cross-fragment priorities are all bypassed.
After #8530 the store build is cached, so this is the main remaining per-open cost on this path — but measured on a warm point-read workload the impact is negligible (0.6 ms / 11k qps with a scheduler still created per open). The concern is IO governance for cold / large scans over shallow-cloned data.
Why it is not a one-liner: the scheduler wraps the resolved (decorated) store, i.e. it bakes the caller's object_store_wrapper into the data path. Caching one scheduler per base on the dataset would freeze the first caller's wrapper and bypass later callers' wrappers for actual reads — the same staleness class #8530's review caught for stores, but on the data path. Reuse needs a design that keys or parameterizes schedulers by wrapper identity (or teaches the scheduler to take the store per call).
Related: the ObjectStoreRegistry holding only weak references (providers.rs) is the underlying reason callers must keep stores alive themselves; a bounded strong-reference LRU in the registry would benefit every from_uri_and_params caller and could subsume the per-dataset cache added in #8530.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in rust/lance/src/dataset/fragment.rs at FileFragment::open_reader_impl and trace the data_file.base_id branch into ScanScheduler::new. Read the scheduler and ObjectStoreRegistry in providers.rs alongside the caching work from #8530. Done means additional-base-path opens reuse scheduling without freezing or bypassing each caller's object_store_wrapper, while preserving configured IO governance.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering, performance
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100