apache / apache/datafusion-comet

Surface Parquet TIMESTAMP_MILLIS overflow as a Spark-faithful exception instead of a raw Arrow error

Open
#5,517 1 comment 0 reactions 1 assignee Claimed by @peterxcli View on GitHub
area:scan bug priority:low
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 4h
Merged PRs (30d)
198

Description

## Context

#5177 makes the Parquet reader's `TIMESTAMP_MILLIS -> microseconds` conversion checked: overflow now raises an error instead of silently wrapping (`parquet_convert_array` in `native/core/src/parquet/parquet_support.rs` uses `try_unary` + `mul_checked`).

The resulting error is a raw `ArrowError::ComputeError("Overflow happened on: * 1000")`, which surfaces to users as a `CometNativeException` rather than what Spark throws on the same input.

## What Spark throws

Spark's vectorized Parquet reader calls `SparkDateTimeUtils.millisToMicros`, which is `Math.multiplyExact`, so it throws an **untyped** `java.lang.ArithmeticException("long overflow")` — independent of ANSI mode:
- https://github.com/apache/spark/blob/v4.2.0/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ParquetVectorUpdaterFactory.java#L817-L833
- https://github.com/apache/spark/blob/v4.2.0/sql/api/src/main/scala/org/apache/spark/sql/catalyst/util/SparkDateTimeUtils.scala#L103-L108

## Why the existing typed variant does not fit

Mapping this to the existing `SparkError::ArithmeticOverflow` would render:

> [ARITHMETIC_OVERFLOW] ... If necessary set "spark.sql.ansi.enabled" to "false" to bypass this error.

That advice is wrong here: this overflow is ANSI-independent (the regression test added in #5177, `ParquetReadSuite."TIMESTAMP_MILLIS overflow fails in native scan"`, asserts the error with ANSI both on and off).

## Proposed work

Add a native error variant that converts to a plain `ArithmeticException("long overflow")` (or the closest Spark-faithful equivalent), with the corresponding `ShimSparkErrorConverter` mappings for each supported Spark version. This aligns with the error-fidelity work in #5169.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.