[Parquet] Prototype FSST encoding
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 2d 18h
- Merged PRs (30d)
- 169
Description
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
- part of https://github.com/apache/arrow-rs/issues/8747
The parquet community is considering adding new encodings to the format.
One proposed encoding is:
* FSST : [Proposal: FSST (Fast Static Symbol Table) Encoding for Parquet](https://docs.google.com/document/d/1g7zgopxeHc5nofJXfc8EEp_HGMaI8g-jFVvNCs2GVA0) ([mailing list thread](https://lists.apache.org/thread/zfcg1cz79wmdp9nlqj1qz6zw9vq37l3k))
**Describe the solution you'd like**
I would like someone(s) to create a branch of arrow-rs 's parquet reader and implement this proposed encoding in the context of parquet to see how it would work.
This would involve likely
1. Adding a new encoding to the metadata: [source link](https://github.com/apache/arrow-rs/blob/78bd20446c1cc54b398e925021933c14cb7784be/parquet/src/basic.rs#L573-L572)
2. Add the appropriate encoder/decoders in parquet ([existing code](https://github.com/apache/arrow-rs/blob/dfc7ef8a3c4ddba1983a16a35d46e004a2e1ad06/parquet/src/encodings/mod.rs#L18-L20))
3. Add some basic tests showing data could be round tripped with these encodings: [examples of similar tets here](https://github.com/apache/arrow-rs/blob/5a384f4c3ccd397dcb8763d89e958da3fa4c666c/parquet/src/arrow/arrow_writer/mod.rs#L1521-L1752)
4. Add some benchmarks: [examples of similar benchmarks](https://github.com/apache/arrow-rs/blob/02fa779a9cb122c5218293be3afb980832701683/parquet/benches/encoding.rs#L98-L97)
There are already several Rust implementations of FSST (both Lance and Vortex) that could probably adapted
- https://crates.io/crates/fsst
- https://crates.io/crates/fsst-rust
**Describe alternatives you've considered**
**Additional context**
Contributor guide
Assessment
This issue has not been assessed yet.