lance-format / lance-format/lance
Investigate better compression for string data like code or websites
@westonpace is already working on this.
Since Jan 8, 2025.
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
Our FSST configuration is not working well for code / websites.
My first guess is that there are just too many opportunities for compression and the symbol table isn't able to take advantage of all of them. In this case we might get better results by chunking in the full zip case (creating multiple symbol tables per page or potentially even per row)
That being said, it also seems like a less than ideal symbol table is being created. I compressed 32MB of code data and ~150 of the 256 symbols are 0 or 1 characters which seems inefficient.
As a short term benefit we could just fallback to a per-row simple block compression when rows are, on average, 2KiB or greater (I picked 2KiB from 5 minutes of googling ideal zip threshold, feel free to investigate more)
This is not strictly needed for the beta milestone but it should be addressed before 2.1 is stabilized since better performance on this type of data is one of our main goals.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.