lance-format / lance-format/lance
Add "rolling compression" to file writer and encodings
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
The compression in 2.1 currently has a problem. We accumulate 8MB of raw data before we run compression. Then we compress the buffer and write a page. However, when the data is highly compressible, this could lead to a very small page being created. Instead of writing a page at that point we should instead wait and add more data.
This will require enhancements to all encoders as it will require them to be able to "save off state" and "resume with more data".
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The issue names the file writer and all encoders but provides no file paths or tests. Start by locating the 2.1 writer and encoder entry points, then trace how the 8MB buffer is compressed and pages are emitted. Done means encoders can save and resume compression state so highly compressible data is accumulated into appropriately sized pages.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100