lance-format / lance-format/lance
Allow dictionary encoding to support more fixed width types
Open
Nobody has claimed this yet.
good first issue
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
Currently the dictionary encoding in 2.1 only applies to some 128-bit fixed-width types. I think this was a combination of a few things:
- At the time, we were not bit-packing indices, and so applying dictionary encoding to smaller types was unlikely to be beneficial
- Estimating cardinality on all columns was expensive
- TPC-H data has a lot of 128-bit decimals that benefit from dictionary encoding so there was an outsized (and unrealistic) benchmarking gain to focusing on 128-bit types.
However, we should at least be considering dictionary encoding for 64-bit types and maybe even 32-bit types.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the dictionary encoding implementation and the logic that limits it to 128-bit fixed-width types. Review how index bit-packing and cardinality estimation affect the decision, then determine the supported 64-bit and 32-bit cases. Done means dictionary encoding is considered for the additional fixed-width types without regressing existing behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data, performance
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100