lance-format / lance-format/lance

Investigate better compression for string data like code or websites

Open
#3,359 0 comments 0 reactions 1 assignee View on GitHub

@westonpace is already working on this.

Since Jan 8, 2025.

A-encoding enhancement
Dominant language
Rust
Stars
7.1k
Forks
852
Avg merge
3d 18h
Merged PRs (30d)
272

Description

Our FSST configuration is not working well for code / websites.

My first guess is that there are just too many opportunities for compression and the symbol table isn't able to take advantage of all of them. In this case we might get better results by chunking in the full zip case (creating multiple symbol tables per page or potentially even per row)

That being said, it also seems like a less than ideal symbol table is being created. I compressed 32MB of code data and ~150 of the 256 symbols are 0 or 1 characters which seems inefficient.

As a short term benefit we could just fallback to a per-row simple block compression when rows are, on average, 2KiB or greater (I picked 2KiB from 5 minutes of googling ideal zip threshold, feel free to investigate more)

This is not strictly needed for the beta milestone but it should be addressed before 2.1 is stabilized since better performance on this type of data is one of our main goals.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.