lance-format / lance-format/lance
Optimize IOPS for opening fragment
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
As found in the unit tests^1, to open a fragment we require:
- 4 IOPS per data file
- (optionally) 1 IOP for a deletion file
If we started storing the size of data files in the manifest, we could skip the first IOP to get the size of the file.
The remaining IOPS are retrieving data from the end of the file. I haven't fully investigated, but I bet they are all within the last block for many file, so we might be able to get them all in a single IOP.
With these two optimizations, that would bring our IOPS per fragment open down to 1 per data file plus (maybe) 1 for the deletion file.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Read the unit-test-referenced code in rust/lance/src/dataset/fragment.rs at lines 2418-2425, then trace how fragment opening reads data and deletion files and how the manifest represents file sizes. Done means reducing fragment-open I/O to the stated target—one IOP per data file plus an optional deletion-file IOP—with unit tests covering the behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- performance
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100