apache / apache/arrow-rs

[Parquet] Prototype FSST encoding

Open
#8,749 6 comments 1 reaction 1 assignee Claimed by @devanbenz View on GitHub
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 18h
Merged PRs (30d)
169

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
- part of https://github.com/apache/arrow-rs/issues/8747

The parquet community is considering adding new encodings to the format.

One proposed encoding is:
* FSST : [Proposal: FSST (Fast Static Symbol Table) Encoding for Parquet](https://docs.google.com/document/d/1g7zgopxeHc5nofJXfc8EEp_HGMaI8g-jFVvNCs2GVA0) ([mailing list thread](https://lists.apache.org/thread/zfcg1cz79wmdp9nlqj1qz6zw9vq37l3k))

**Describe the solution you'd like**

I would like someone(s) to create a branch of arrow-rs 's parquet reader and implement this proposed encoding in the context of parquet to see how it would work.

This would involve likely
1. Adding a new encoding to the metadata: [source link](https://github.com/apache/arrow-rs/blob/78bd20446c1cc54b398e925021933c14cb7784be/parquet/src/basic.rs#L573-L572)
2. Add the appropriate encoder/decoders in parquet ([existing code](https://github.com/apache/arrow-rs/blob/dfc7ef8a3c4ddba1983a16a35d46e004a2e1ad06/parquet/src/encodings/mod.rs#L18-L20))
3. Add some basic tests showing data could be round tripped with these encodings: [examples of similar tets here](https://github.com/apache/arrow-rs/blob/5a384f4c3ccd397dcb8763d89e958da3fa4c666c/parquet/src/arrow/arrow_writer/mod.rs#L1521-L1752)
4. Add some benchmarks: [examples of similar benchmarks](https://github.com/apache/arrow-rs/blob/02fa779a9cb122c5218293be3afb980832701683/parquet/benches/encoding.rs#L98-L97)

There are already several Rust implementations of FSST (both Lance and Vortex) that could probably adapted
- https://crates.io/crates/fsst
- https://crates.io/crates/fsst-rust

**Describe alternatives you've considered**

**Additional context**

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.