lance-format / lance-format/lance
Investigate FSST on large values
Open
Nobody has claimed this yet.
A-encoding
enhancement
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
I tried applying FSST to large values (code, websites) and did not get very good compression. We should investigate why that is.
- Perhaps the problem is that we are only creating one symbol table per 8MB page, we could create a symbol table per value for these large values.
- Perhaps there is a bug in the way we are constructing the symbol table, we should compare performance with other FSST implementations
- Perhaps a single 1-byte symbol table is not large enough for describing all the patterns in code or websites
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No file, test, or entry point is named. Start by locating the Rust FSST implementation and reproducing compression on large code and website values; compare it with other FSST implementations and evaluate page-level versus per-value symbol tables and symbol-table sizing. Done means the cause of the poor compression is identified and a measured direction is agreed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100