apache / apache/datafusion-comet
Surface Parquet TIMESTAMP_MILLIS overflow as a Spark-faithful exception instead of a raw Arrow error
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 198
Description
## Context
#5177 makes the Parquet reader's `TIMESTAMP_MILLIS -> microseconds` conversion checked: overflow now raises an error instead of silently wrapping (`parquet_convert_array` in `native/core/src/parquet/parquet_support.rs` uses `try_unary` + `mul_checked`).
The resulting error is a raw `ArrowError::ComputeError("Overflow happened on: * 1000")`, which surfaces to users as a `CometNativeException` rather than what Spark throws on the same input.
## What Spark throws
Spark's vectorized Parquet reader calls `SparkDateTimeUtils.millisToMicros`, which is `Math.multiplyExact`, so it throws an **untyped** `java.lang.ArithmeticException("long overflow")` — independent of ANSI mode:
- https://github.com/apache/spark/blob/v4.2.0/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ParquetVectorUpdaterFactory.java#L817-L833
- https://github.com/apache/spark/blob/v4.2.0/sql/api/src/main/scala/org/apache/spark/sql/catalyst/util/SparkDateTimeUtils.scala#L103-L108
## Why the existing typed variant does not fit
Mapping this to the existing `SparkError::ArithmeticOverflow` would render:
> [ARITHMETIC_OVERFLOW] ... If necessary set "spark.sql.ansi.enabled" to "false" to bypass this error.
That advice is wrong here: this overflow is ANSI-independent (the regression test added in #5177, `ParquetReadSuite."TIMESTAMP_MILLIS overflow fails in native scan"`, asserts the error with ANSI both on and off).
## Proposed work
Add a native error variant that converts to a plain `ArithmeticException("long overflow")` (or the closest Spark-faithful equivalent), with the corresponding `ShimSparkErrorConverter` mappings for each supported Spark version. This aligns with the error-fidelity work in #5169.
Contributor guide
Assessment
This issue has not been assessed yet.