apache / apache/datafusion-comet
Native Parquet schema-on-read diverges from Spark for ANSI interval targets
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 198
Description
### Describe the bug
When an explicit Parquet read schema requests an ANSI interval type that does not match the file column's logical type, Comet's native scan differs from Spark in two directions:
- A Parquet `DATE` column requested as `INTERVAL YEAR TO MONTH` succeeds in Spark but fails in Comet.
- A Parquet `INT` column requested as `INTERVAL DAY TO SECOND` fails in Spark but succeeds in Comet, reinterpreting the integers as microseconds.
The second case is a correctness gap because Comet returns rows for a read that Spark rejects with `PARQUET_COLUMN_DATA_TYPE_MISMATCH`.
### Steps to reproduce
Write a one-column Parquet file, then read it with an explicit schema. Run each read once with Comet disabled for the Spark reference and once with the native Parquet scan enabled.
```scala
spark.conf.set("spark.sql.sources.useV1SourceList", "parquet")
spark.sql("SELECT CAST(DATE '2020-01-01' AS DATE) AS c")
.write.parquet(datePath)
spark.read.schema("c INTERVAL YEAR TO MONTH")
.parquet(datePath).collect()
spark.sql("SELECT CAST(1 AS INT) AS c")
.write.parquet(intPath)
spark.read.schema("c INTERVAL DAY TO SECOND")
.parquet(intPath).collect()
```
The full observed matrix on Spark 4.1 is:
| File column | Requested schema | Spark | Comet |
| --- | --- | --- | --- |
| `date` | `interval year to month` | succeeds (raw INT32 value interpreted as total months) | fails: `expected Interval(YearMonth) but found Date32` |
| `int` | `interval day to second` | `PARQUET_COLUMN_DATA_TYPE_MISMATCH` | succeeds; values are reinterpreted as microseconds |
| `timestamp` | `interval day to second` | error | error |
| `long` | `interval year to month` | error | error |
### Expected behavior
Comet should match Spark's `ParquetVectorUpdaterFactory` compatibility rules:
- Accept INT32, including `DATE`, for `YearMonthIntervalType` and interpret the raw value as total months.
- Accept INT64 for `DayTimeIntervalType`.
- Reject INT32/`INT` for `DayTimeIntervalType` with Spark-compatible schema-mismatch behavior.
- Continue rejecting the timestamp-to-day-time and long-to-year-month control cases.
### Additional context
The compatibility decision belongs in the native Parquet schema adapter. Conversion then reaches `parquet_convert_array`, whose Arrow `can_cast_types` path accepts numeric-to-`Duration`, while its fallback can leave `Date32` unconverted for an `Interval(YearMonth)` target.
- Schema adapter: https://github.com/apache/datafusion-comet/blob/2963b0871436766fd14eb15fbe08f57ad50af30e/native/core/src/parquet/schema_adapter.rs#L773-L829
- Array conversion: https://github.com/apache/datafusion-comet/blob/2963b0871436766fd14eb15fbe08f57ad50af30e/native/core/src/parquet/parquet_support.rs#L166-L260
- PR: https://github.com/apache/datafusion-comet/pull/5161
- Review finding: https://github.com/apache/datafusion-comet/pull/5161#discussion_r3690464239
- Related generic schema-mismatch work: #3720 and #4297
This issue is limited to schema-on-read compatibility for ANSI interval targets. Matching-schema interval scans are covered by #5060 / #5161. Add focused coverage beside the existing primitive schema-conversion cases in `ParquetReadSuite`.
Contributor guide
Assessment
This issue has not been assessed yet.