lance-format / lance-format/lance
Better determination of FSST output buffer size
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
Currently, the FSST output buffer is initialized to 8 * compressed_length. This should be safe (no FSST symbol is more than 8 bytes I think) but it is slow and excessive. We can store the original length of the string as part of the encoded page and use that to size the output buffer exactly.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the FSST encoder and the encoded-page layout described in the issue, then trace how compressed strings determine their output buffer size. Store the original string length in the encoded page and use it for exact allocation; done means the buffer is no longer sized as 8 times the compressed length and existing FSST behavior remains correct.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering, performance
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100