apache / apache/arrow-rs

Optimize `compare_greater` for faster parquet writer

Open
#8,792 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 18h
Merged PRs (30d)
169

Description

@jhorstmann notes:

>
> I looked at this function in #7916 and noted
>
> I looked into also specializing the `get_min_max` function by moving the logical type checks outside of the loop, but that did not bring any further measurable improvement.
>
> Maybe trying again to vectorize that code could be worthwhile.

_Originally posted by @jhorstmann in [#8783](https://github.com/apache/arrow-rs/issues/8783#issuecomment-3490077327)_

Contributor guide

Open the contributing guide

Research direction

Locate `compare_greater` in the parquet writer and inspect the `get_min_max` loop and its logical type checks. Compare any vectorization attempt with the current implementation using the project's existing performance measurements; done means a measurable writer speedup without changing output.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering, performance
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.