lance-format / lance-format/lance
Optimize local fs listing of transactions
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
In #3882 we are making conflict resolution a little slower for local filesystems. This is because we are using ObjectStore's list to list all the manifests. That implementation will call stat on each entry, even if it's a file path we don't care about.
Instead we can use a similar strategy we have in:
- Create an iterator of
ManifestLocationjust from thereaddirresults (will be missinge_tagandsize). - Collect just the entries we care about
- Then call
statof each of those in paralel to fill ine_tagandsize.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the analogous strategy in rust/lance-table/src/io/commit.rs around line 333, then trace the local filesystem transaction listing that uses ObjectStore to enumerate manifests. The change is complete when relevant ManifestLocation entries come from readdir results and only those entries are stat'ed in parallel to fill e_tag and size.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Refactor
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100