apache / apache/datafusion-comet

Native Parquet schema-on-read diverges from Spark for ANSI interval targets

Open
#5,188 1 comment 0 reactions 1 assignee Claimed by @peterxcli View on GitHub
area:scan bug priority:medium
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 4h
Merged PRs (30d)
198

Description

### Describe the bug

When an explicit Parquet read schema requests an ANSI interval type that does not match the file column's logical type, Comet's native scan differs from Spark in two directions:

- A Parquet `DATE` column requested as `INTERVAL YEAR TO MONTH` succeeds in Spark but fails in Comet.
- A Parquet `INT` column requested as `INTERVAL DAY TO SECOND` fails in Spark but succeeds in Comet, reinterpreting the integers as microseconds.

The second case is a correctness gap because Comet returns rows for a read that Spark rejects with `PARQUET_COLUMN_DATA_TYPE_MISMATCH`.

### Steps to reproduce

Write a one-column Parquet file, then read it with an explicit schema. Run each read once with Comet disabled for the Spark reference and once with the native Parquet scan enabled.

```scala
spark.conf.set("spark.sql.sources.useV1SourceList", "parquet")

spark.sql("SELECT CAST(DATE '2020-01-01' AS DATE) AS c")
.write.parquet(datePath)
spark.read.schema("c INTERVAL YEAR TO MONTH")
.parquet(datePath).collect()

spark.sql("SELECT CAST(1 AS INT) AS c")
.write.parquet(intPath)
spark.read.schema("c INTERVAL DAY TO SECOND")
.parquet(intPath).collect()
```

The full observed matrix on Spark 4.1 is:

| File column | Requested schema | Spark | Comet |
| --- | --- | --- | --- |
| `date` | `interval year to month` | succeeds (raw INT32 value interpreted as total months) | fails: `expected Interval(YearMonth) but found Date32` |
| `int` | `interval day to second` | `PARQUET_COLUMN_DATA_TYPE_MISMATCH` | succeeds; values are reinterpreted as microseconds |
| `timestamp` | `interval day to second` | error | error |
| `long` | `interval year to month` | error | error |

### Expected behavior

Comet should match Spark's `ParquetVectorUpdaterFactory` compatibility rules:

- Accept INT32, including `DATE`, for `YearMonthIntervalType` and interpret the raw value as total months.
- Accept INT64 for `DayTimeIntervalType`.
- Reject INT32/`INT` for `DayTimeIntervalType` with Spark-compatible schema-mismatch behavior.
- Continue rejecting the timestamp-to-day-time and long-to-year-month control cases.

### Additional context

The compatibility decision belongs in the native Parquet schema adapter. Conversion then reaches `parquet_convert_array`, whose Arrow `can_cast_types` path accepts numeric-to-`Duration`, while its fallback can leave `Date32` unconverted for an `Interval(YearMonth)` target.

- Schema adapter: https://github.com/apache/datafusion-comet/blob/2963b0871436766fd14eb15fbe08f57ad50af30e/native/core/src/parquet/schema_adapter.rs#L773-L829
- Array conversion: https://github.com/apache/datafusion-comet/blob/2963b0871436766fd14eb15fbe08f57ad50af30e/native/core/src/parquet/parquet_support.rs#L166-L260
- PR: https://github.com/apache/datafusion-comet/pull/5161
- Review finding: https://github.com/apache/datafusion-comet/pull/5161#discussion_r3690464239
- Related generic schema-mismatch work: #3720 and #4297

This issue is limited to schema-on-read compatibility for ANSI interval targets. Matching-schema interval scans are covered by #5060 / #5161. Add focused coverage beside the existing primitive schema-conversion cases in `ParquetReadSuite`.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.