lance-format / lance-format/lance
Notes from reveiw of 2.1 encodings
Open
Nobody has claimed this yet.
enhancement
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
The following are notes I took while reviewing 2.1 encodings in preparation for making 2.1 stable.
- The binary mini block encoding does not compress offsets and it should
- High priority: offsets are likely very compressible (delta and bit packing in particular)
- The binary mini block encoding should allow general compression to be applied to the values buffer
- Low priority: mini-block level compression is probably fine
- Mini block inline bitpacking should allow the inner compression to be specified (e.g. bitpacking followed by general compression should be possible)
- Low priority: mini-block level compression is probably fine
- The general-per-value encoding should support setting the compression level
- High priority: this is an interesting config that should be exposed
- The general-per-value encoding should defer to the binary encoding
- Medium priority: If we don't do this we won't have compression of the offsets, but offsets compression isn't terribly useful in per-value situations
- Byte stream split could support underlying compression
- Low priority: mini-block level compression is probably fine
- We are not compressing rep/def in miniblock
- High priority: rep/def is quite large if not compressed at all, this is wasteful
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the implementations for binary mini block, general-per-value, byte stream split, and repetition/definition-level encodings. Review how offsets, values, compression configuration, and rep/def data are handled, then prioritize the high-priority items in the note. Done means the selected compression behaviors are implemented and verified by tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100