lance-format / lance-format/lance
Align on 64 bits for number of values in 2.1 encodings
Open
@HaochengLIU is already working on this.
Since Jul 21, 2025.
enhancement
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
There are a few spots in the 2.1 encodings where we are splitting into 32-bit and 64-bit logic that don't seem necessary.
- When we dictionary encode arrays we currently use 32-bit offsets for fixed types and string/binary and 64-bit offset for large-string/large-binary. The offsets are based on the number of unique values. There is no reason for these to be different. In fact, it seems more likely we would have more than 2^32 values with fixed values than anything else.
- In the variable encoder block compressor we store the
number of valuesin a field of widthbits_per_offset. The number of values is not related to the number of bits per offset. It should be 8 bytes in both cases (small and large binary) - Possibly others?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.