Optimize `compare_greater` for faster parquet writer
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 2d 18h
- Merged PRs (30d)
- 169
Description
@jhorstmann notes:
>
> I looked at this function in #7916 and noted
>
> I looked into also specializing the `get_min_max` function by moving the logical type checks outside of the loop, but that did not bring any further measurable improvement.
>
> Maybe trying again to vectorize that code could be worthwhile.
_Originally posted by @jhorstmann in [#8783](https://github.com/apache/arrow-rs/issues/8783#issuecomment-3490077327)_
Contributor guide
Research direction
Locate `compare_greater` in the parquet writer and inspect the `get_min_max` loop and its logical type checks. Compare any vectorization attempt with the current implementation using the project's existing performance measurements; done means a measurable writer speedup without changing output.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering, performance
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100