apache / apache/arrow

[C++][Dataset] ORC predicate pushdown: full operator and type coverage

Open
#49,362 0 comments 0 reactions 0 assignees View on GitHub
Component: C++
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Summary

Part 3 of ORC predicate pushdown (#48986). Depends on #49361.

Extend the initial INT32/INT64 greater-than implementation to cover all comparison operators, logical operators, set operations, and null handling.

### Operator coverage

The guarantee-based approach means most operators work automatically once the guarantee expression is correct. The work here is:
1. Ensuring guarantee expressions handle null semantics correctly for each operator class
2. Adding test coverage for each operator
3. Handling edge cases specific to certain operators (e.g., IN with mixed types)

| Category | Operators | Notes |
|----------|-----------|-------|
| Comparison | `>`, `>=`, `<`, `<=`, `==`, `!=` | All work via `SimplifyWithGuarantee()` with min/max range guarantees |
| Logical | `AND`, `OR`, `NOT` | Compound predicates; Arrow's simplifier handles these given correct per-field guarantees |
| Set | `IN` | Range intersection: if all IN values fall outside [min, max], skip stripe |
| Null | `IS NULL`, `IS NOT NULL` | Use `hasNull()` and `getNumberOfValues() == 0` from ORC stats |

### Future type extensions

This sub-issue covers operators for INT32/INT64. Extending to additional types is a follow-up:

| Type | Key concern |
|------|------------|
| DOUBLE, FLOAT | NaN in statistics makes range unusable; ±Inf are valid bounds |
| STRING | ORC may truncate long strings in statistics; collation/encoding assumptions |
| DATE | int32 days since epoch — straightforward |
| TIMESTAMP | Unit conversion (ORC millis + sub-millis nanos → Arrow nanos) |
| DECIMAL | Scale/precision must match between stats and field type |

### Tests

- Each comparison operator individually (>, >=, <, <=, ==, !=)
- AND compound predicate (both conditions must hold)
- OR compound predicate (either condition suffices)
- NOT operator
- IN operator with values inside/outside stripe range
- IS NULL on stripe with/without nulls
- IS NOT NULL on all-null stripe
- Compound: `(id > 100 AND id < 200) OR id == 500`
- Unsupported type in predicate → conservative include (no skip)

### Component(s)

C++

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.