apache / apache/arrow

[C++][Dataset] Add OrcFileFragment with stripe filtering and predicate pushdown

Open
#49,361 0 comments 0 reactions 0 assignees View on GitHub
Component: C++
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Summary

Sub Issue 2 of ORC predicate pushdown (#48986)

Add the dataset-layer infra for ORC predicate pushdown, modeled after `ParquetFileFragment`. This effectively connects ORC stripe statistics to Arrow's expression API in order to skip stripes at scan time.

### Changes

OrcFileFragment` (inheriting `FileFragment`). Summarized below as a stripped-down header:
```cpp
// For context: the Parquet equivalent has the same pattern with
// row groups where ORC has stripes.

class OrcFileFragment : public FileFragment {
public:
// Selected stripes, or std::nullopt if all stripes are selected.
const std::optional>& stripes() const;

// Return a new fragment representing only the specified stripes.
Result> Subset(std::vector stripe_ids);

private:
// Convert ORC stripe statistics to a guarantee expression:
// field >= min AND field <= max [OR is_null(field)]
// Uses the adapter-layer GetStripeColumnStatistics() from #49360.
static std::optional DeriveFieldGuarantee(
const adapters::orc::OrcColumnStatistics& stats,
const Field& field);

// For each stripe, build guarantee from statistics and call
// SimplifyWithGuarantee(predicate, guarantee).
// Returns per-stripe simplified expressions.
Result> TestStripes(
compute::Expression predicate);

// Wrapper over TestStripes(): returns stripe indices where
// IsSatisfiable() is true (i.e. stripes that might match).
Result> FilterStripes(compute::Expression predicate);

std::optional> stripes_;
// Per-stripe combined guarantee expressions (cached)
std::vector statistics_expressions_;
// Tracks which columns have had statistics processed
std::vector statistics_expressions_complete_;
std::mutex physical_schema_mutex_;

friend class OrcFileFormat;
};
```

Updated `OrcFileFormat`. Summarized:
```cpp
class OrcFileFormat : public FileFormat {
public:
// --- existing API (unchanged) ---
Result IsSupported(const FileSource& source) const override;
Result> Inspect(const FileSource& source) const override;

// --- UPDATED: now calls FilterStripes() to skip stripes that fail the predicate,
// then streams matching stripes via Seek() + NextStripeReader()
// (not ReadStripes() which materializes to Table) ---
Result ScanBatchesAsync(
const std::shared_ptr& options,
const std::shared_ptr& file) const override;
};
```

**Infrastructure:**

- Lazy evaluation: only process statistics for fields referenced in the predicate
- Statistics caching: avoid re-parsing stripe statistics on repeated access
- Thread safety: `physical_schema_mutex_` guards cached metadata (same pattern as Parquet)
- Feature flag: `ARROW_ORC_DISABLE_PREDICATE_PUSHDOWN=1` to bypass filtering
- ORC column index mapping: Arrow field index → ORC depth-first pre-order column ID (col 0 = root struct, col 1 = first top-level field, etc.), using `GetORCType()` from #49360

**Initial type support:** INT32, INT64
- INT32 overflow protection: if liborc int64 stats exceed INT32 bounds, conservatively include the stripe

### Tests

Dataset-layer tests in `file_orc_test.cc`:
- Stripe filtering with selective predicates (verify correct stripes skipped/kept)
- Greater-than predicate on multi-stripe file
- Feature flag disables pushdown
- Missing statistics → conservative include
- INT32 overflow → conservative include
- Multi-stripe file with known per-stripe value ranges
- Explicit stripe selection via `Subset()`

### Component(s)

C++

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.