apache / apache/datafusion-comet
Checked TIMESTAMP_MILLIS overflow for nested fields and nested-predicate scans is blocked on DataFusion nested-field pruning
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 198
Description
## Context
#5177 makes the native Parquet reader's `TIMESTAMP_MILLIS -> microseconds` conversion checked for **top-level** columns, matching Spark's `millisToMicros` (`Math.multiplyExact`). Two cases intentionally keep the pre-existing safe-cast behavior (overflow -> NULL) instead of erroring like Spark:
1. **Nested fields** (struct children, list/map elements): the conversion inside `parquet_convert_struct_to_struct` / list / map recursion is unchecked.
2. **Scans whose data filters reference nested fields** (`GetStructField`): the whole scan falls back to the safe cast (`SparkParquetOptions.checked_timestamp_overflow = false`, set in `init_datasource_exec`).
## Why
Spark only avoids the overflow error for filtered-out values through row-group statistics pruning (`ParquetFilters` supports nested column predicates). DataFusion currently can neither:
- prune nested-field predicates — `datafusion-pruning` says "PruningPredicate does not support pruning on nested fields yet" (`pruning_predicate.rs`), nor
- evaluate them as Parquet row filters — `can_expr_be_pushed_down_with_schemas` in `datafusion-datasource-parquet/src/row_filter.rs` classifies struct columns as non-pushable.
So a checked conversion on these paths would decode row groups Spark prunes and fail queries Spark answers with zero rows (see the review discussion on #5177: `ts IN (...)`, `ts <=> ...`, and `s.ts < ...` reproducers).
## Remaining divergence vs Spark
- A nested-predicate scan that Spark *fails* to prune (mixed row-group statistics) throws `ArithmeticException` in Spark but returns NULL for overflowing top-level values in Comet.
- A direct read of an overflowing nested `TIMESTAMP_MILLIS` field throws in Spark but returns NULL in Comet (pre-existing behavior on main).
## Proposed work
Once DataFusion supports nested-field pruning (and ideally nested row filters), remove the `checked_timestamp_overflow` fallback and extend the checked conversion to nested fields, plus extend the millisecond-domain predicate rewrite in `SparkPhysicalExprAdapter` to `GetStructField`-wrapped conversions.
Regression coverage lives in `ParquetReadSuite`:
- "TIMESTAMP_MILLIS overflow fails in native scan"
- "TIMESTAMP_MILLIS overflow rows skipped by filter pruning do not fail"
Related: #5517 (error fidelity for the overflow exception).
Contributor guide
Assessment
This issue has not been assessed yet.