apache / apache/arrow-rs

[Parquet] Validate the ALP sampling heuristic

Open
#10,646 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 14h
Merged PRs (30d)
167

Description

### Is your feature request related to a problem or challenge?

The ALP encoder samples input values to select candidate exponent and factor pairs. We should verify that this heuristic produces compression reasonably close to evaluating every valid pair.

Pathological inputs where sampling selects significantly worse parameters should become deterministic regression cases.

This depends on #9372 and should be worked on after that PR is merged.

Contributor guide

Open the contributing guide

Research direction

After #9372 is merged, locate the ALP encoder and its sampling heuristic. Compare sampled candidate selection with evaluation of every valid exponent and factor pair, then identify pathological inputs where the result is significantly worse. Add deterministic regression cases demonstrating the expected compression-quality behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering, performance, testing
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.