lance-format / lance-format/lance
Cache fragment metadata in-memory
Open
Nobody has claimed this yet.
enhancement
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
For Take we currently have to perform one set of IOs to read fragment metadata, and then we read the actual data.
We could significantly improve the e2e latency by caching the fragment metadata. We would also need a reliable mechanism to "warm up" this metadata cache.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the Take path that reads fragment metadata and then reads the actual data. Investigate how metadata could be retained in memory and what reliable cache warm-up would require. Done means the extra metadata I/O is avoided without making cache behavior unreliable.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100