apache / apache/datafusion-comet

Checked TIMESTAMP_MILLIS overflow for nested fields and nested-predicate scans is blocked on DataFusion nested-field pruning

Open
#5,553 1 comment 0 reactions 1 assignee Claimed by @peterxcli View on GitHub
area:scan bug correctness priority:critical
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 4h
Merged PRs (30d)
198

Description

## Context

#5177 makes the native Parquet reader's `TIMESTAMP_MILLIS -> microseconds` conversion checked for **top-level** columns, matching Spark's `millisToMicros` (`Math.multiplyExact`). Two cases intentionally keep the pre-existing safe-cast behavior (overflow -> NULL) instead of erroring like Spark:

1. **Nested fields** (struct children, list/map elements): the conversion inside `parquet_convert_struct_to_struct` / list / map recursion is unchecked.
2. **Scans whose data filters reference nested fields** (`GetStructField`): the whole scan falls back to the safe cast (`SparkParquetOptions.checked_timestamp_overflow = false`, set in `init_datasource_exec`).

## Why

Spark only avoids the overflow error for filtered-out values through row-group statistics pruning (`ParquetFilters` supports nested column predicates). DataFusion currently can neither:

- prune nested-field predicates — `datafusion-pruning` says "PruningPredicate does not support pruning on nested fields yet" (`pruning_predicate.rs`), nor
- evaluate them as Parquet row filters — `can_expr_be_pushed_down_with_schemas` in `datafusion-datasource-parquet/src/row_filter.rs` classifies struct columns as non-pushable.

So a checked conversion on these paths would decode row groups Spark prunes and fail queries Spark answers with zero rows (see the review discussion on #5177: `ts IN (...)`, `ts <=> ...`, and `s.ts < ...` reproducers).

## Remaining divergence vs Spark

- A nested-predicate scan that Spark *fails* to prune (mixed row-group statistics) throws `ArithmeticException` in Spark but returns NULL for overflowing top-level values in Comet.
- A direct read of an overflowing nested `TIMESTAMP_MILLIS` field throws in Spark but returns NULL in Comet (pre-existing behavior on main).

## Proposed work

Once DataFusion supports nested-field pruning (and ideally nested row filters), remove the `checked_timestamp_overflow` fallback and extend the checked conversion to nested fields, plus extend the millisecond-domain predicate rewrite in `SparkPhysicalExprAdapter` to `GetStructField`-wrapped conversions.

Regression coverage lives in `ParquetReadSuite`:
- "TIMESTAMP_MILLIS overflow fails in native scan"
- "TIMESTAMP_MILLIS overflow rows skipped by filter pruning do not fail"

Related: #5517 (error fidelity for the overflow exception).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.