apache / apache/arrow-rs

Implement ALP encoding

Open
#10,541 1 comment 2 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 14h
Merged PRs (30d)
167

Description

### Is your feature request related to a problem or challenge?

- Follow on to https://github.com/apache/arrow-rs/issues/8748

Parquet has added a new encoding for floating point numbers: ALP

The format change is here
- https://github.com/apache/parquet-format/pull/557

### Describe the solution you'd like

We should implement and support this new encoding in arrow-rs

### Describe alternatives you've considered

@sdf-jkl and @devanbenz have a PR almost ready to go in
- https://github.com/apache/arrow-rs/pull/9372

### Additional context

I am filing this issue so I can close the prototype one which has more backstory
- https://github.com/apache/arrow-rs/issues/8748

### Subtasks
- [ ] #10642
- [ ] #10645
- [ ] #10646
- [ ] #10644
- [ ] #10643

Contributor guide

Open the contributing guide

Research direction

Start by reading the Parquet format change in PR 557 and the prototype context in issue 8748, then review the nearly ready implementation in PR 9372. The issue lists subtasks #10642, #10645, #10646, #10644, and #10643; done means ALP encoding is implemented and supported in arrow-rs with those subtasks addressed.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.