[C++][Dataset] Add OrcFileFragment with stripe filtering and predicate pushdown
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
### Summary
Sub Issue 2 of ORC predicate pushdown (#48986)
Add the dataset-layer infra for ORC predicate pushdown, modeled after `ParquetFileFragment`. This effectively connects ORC stripe statistics to Arrow's expression API in order to skip stripes at scan time.
### Changes
OrcFileFragment` (inheriting `FileFragment`). Summarized below as a stripped-down header:
```cpp
// For context: the Parquet equivalent has the same pattern with
// row groups where ORC has stripes.
class OrcFileFragment : public FileFragment {
public:
// Selected stripes, or std::nullopt if all stripes are selected.
const std::optional>& stripes() const;
// Return a new fragment representing only the specified stripes.
Result> Subset(std::vector stripe_ids);
private:
// Convert ORC stripe statistics to a guarantee expression:
// field >= min AND field <= max [OR is_null(field)]
// Uses the adapter-layer GetStripeColumnStatistics() from #49360.
static std::optional DeriveFieldGuarantee(
const adapters::orc::OrcColumnStatistics& stats,
const Field& field);
// For each stripe, build guarantee from statistics and call
// SimplifyWithGuarantee(predicate, guarantee).
// Returns per-stripe simplified expressions.
Result> TestStripes(
compute::Expression predicate);
// Wrapper over TestStripes(): returns stripe indices where
// IsSatisfiable() is true (i.e. stripes that might match).
Result> FilterStripes(compute::Expression predicate);
std::optional> stripes_;
// Per-stripe combined guarantee expressions (cached)
std::vector statistics_expressions_;
// Tracks which columns have had statistics processed
std::vector statistics_expressions_complete_;
std::mutex physical_schema_mutex_;
friend class OrcFileFormat;
};
```
Updated `OrcFileFormat`. Summarized:
```cpp
class OrcFileFormat : public FileFormat {
public:
// --- existing API (unchanged) ---
Result IsSupported(const FileSource& source) const override;
Result> Inspect(const FileSource& source) const override;
// --- UPDATED: now calls FilterStripes() to skip stripes that fail the predicate,
// then streams matching stripes via Seek() + NextStripeReader()
// (not ReadStripes() which materializes to Table) ---
Result ScanBatchesAsync(
const std::shared_ptr& options,
const std::shared_ptr& file) const override;
};
```
**Infrastructure:**
- Lazy evaluation: only process statistics for fields referenced in the predicate
- Statistics caching: avoid re-parsing stripe statistics on repeated access
- Thread safety: `physical_schema_mutex_` guards cached metadata (same pattern as Parquet)
- Feature flag: `ARROW_ORC_DISABLE_PREDICATE_PUSHDOWN=1` to bypass filtering
- ORC column index mapping: Arrow field index → ORC depth-first pre-order column ID (col 0 = root struct, col 1 = first top-level field, etc.), using `GetORCType()` from #49360
**Initial type support:** INT32, INT64
- INT32 overflow protection: if liborc int64 stats exceed INT32 bounds, conservatively include the stripe
### Tests
Dataset-layer tests in `file_orc_test.cc`:
- Stripe filtering with selective predicates (verify correct stripes skipped/kept)
- Greater-than predicate on multi-stripe file
- Feature flag disables pushdown
- Missing statistics → conservative include
- INT32 overflow → conservative include
- Multi-stripe file with known per-stripe value ranges
- Explicit stripe selection via `Subset()`
### Component(s)
C++
Contributor guide
Assessment
This issue has not been assessed yet.