lance-format / lance-format/lance
Make sure ScanScheduler works well with multiple object stores
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
Currently, the ScanScheduler is designed to work with a single object store. When multiple object stores are involved, this can lead to performance issues and just bad defaults for various configs because the configs assumes 1 object store connection.
In the multi-base implementation, the quick workaround I did was to create one scheduler per object store, but I remember when I discussed with @westonpace during that time, the intended design was that there should only be a single scheduler, and it should instead reference multiple object stores.
This is combing back again in the external blob case @Xuanwo did in blob v2, as external blobs can have different prefixes thus using different object stores, so we need a way to manage the scan scheduler with multiple object stores.
Creating this issue to track this.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing ScanScheduler and the multi-base implementation described in the issue, then compare the one-scheduler-per-object-store workaround with the external blob v2 case. Done means a single scheduler can manage multiple object stores and no longer assumes one object-store connection when choosing configuration defaults.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100