apache / apache/arrow-rs

[EPIC] [Parquet] Implement new Parquet encodings in arrow-rs

Open
#8,747 1 comment 6 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 16h
Merged PRs (30d)
168

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
The parquet community is considering adding new encodings to the format. Part of adding new encodings is evaluating them on both real world datasets and ensuring they are efficiently implementable in multiple languages

Clearly, I would like Rust to be one of those languages

This epic tracks the work for adding new encodings to Arrow-rs if/when they are added to the parquet spec

- [x] https://github.com/apache/arrow-rs/issues/8748
- [ ] https://github.com/apache/arrow-rs/issues/8749

Contributor guide

Open the contributing guide

Research direction

This is an epic tracking new Parquet encodings in arrow-rs, with completed work in issue 8748 and remaining work in issue 8749. Start by reading issue 8749 and the Parquet specification context it references. Done means the remaining encoding work is implemented and the associated evaluation is complete.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.