apache / apache/arrow

ORC Predicate Pushdown

Open
#48,986 2 comments 0 reactions 1 assignee Claimed by @cbb330 View on GitHub
Component: Python Type: enhancement
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the enhancement requested

Arrow's ORC reader already supports column projection (reading only selected columns), but lacks row-level predicate pushdown. Currently, filtering rows from ORC files requires:
1. Reading all rows from selected columns (all stripes)
2. Applying filters post-read using Arrow compute

This is inefficient for large ORC files where only a small subset of rows match the filter criteria. ORC files store min/max statistics at the stripe level, which can be used to skip entire stripes that cannot contain matching rows, and avoids I/O for data that will be filtered out anyway.

### Use Cases

1. Efficiently query large ORC datasets with selective predicates
2. Enable predicate pushdown for Iceberg tables stored in ORC format
3. Match the filtering capabilities already available for Parquet files

### Component(s)

Python, C++

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.