apache / apache/datafusion-comet
Parquet Variant annotation validation is bypassed when reading as an ordinary struct
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 198
Description
### Describe the bug
With Comet enabled on Spark 4.1.3, reading a Parquet VARIANT-annotated field as an ordinary `struct` does not raise the error Spark requires when `SQLConf.PARQUET_IGNORE_VARIANT_ANNOTATION` is false.
Found while restoring the Variant shredding suites for #5569. The existing Spark test `ParquetVariantShreddingSuite / variant logical type annotation - ignore variant annotation` fails with:
```text
Expected exception org.apache.spark.SparkException to be thrown, but no exception was thrown
(ParquetVariantShreddingSuite.scala:277)
```
[CI log](https://github.com/rich7420/datafusion-comet/actions/runs/34022223542/job/101462446982). This is an observed missing exception, not evidence of silent-null corruption in this case.
### Steps to reproduce
Use Spark v4.1.3 with the Comet patch, removing the `IgnoreCometSuite` mixin from `ParquetVariantShreddingSuite`. The failure was observed in the [fork Spark 4.1 CI job](https://github.com/rich7420/datafusion-comet/actions/runs/34022223542/job/101462446982), at commit `00fa408852c5fa1714fdd00b3de8212c2f2394ce`, based on Comet `7e1984399eb887cd13109698ee55cf2ce150f849`.
Run the suite with Comet enabled through the Spark SQL test harness. A focused command after the documented Comet build/install and Spark patch setup is:
```sh
NOLINT_ON_COMPILE=true ENABLE_COMET=true ENABLE_COMET_ONHEAP=true \
build/sbt 'sql/testOnly org.apache.spark.sql.execution.datasources.parquet.ParquetVariantShreddingSuite'
```
The CI reproduction used the full `sql_core-1` test selection, not this focused command. Environment: Spark 4.1.3, JDK 17, Linux.
The [existing test](https://github.com/apache/spark/blob/v4.1.3/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetVariantShreddingSuite.scala#L233-L290) writes annotated Variant values at the top level and inside structs, arrays and maps. It checks both values of `PARQUET_IGNORE_VARIANT_ANNOTATION`. The `true` arm succeeds; the `false` arm reaches `struct_result.collect()` without throwing.
### Expected behavior
Match Spark's reader policy: reject the incompatible struct read when Variant annotations must be respected, and preserve the successful struct read when ignoring annotations is explicitly enabled.
Spark expects a `SparkException` caused by an `AnalysisException` with condition `_LEGACY_ERROR_TEMP_3071` and an `Invalid Spark read type` message.
### Additional context
Source tracing points to a different boundary from #4084's shredded-Variant fallback:
- Spark's [ParquetToSparkSchemaConverter](https://github.com/apache/spark/blob/v4.1.3/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetSchemaConverter.scala#L393-L410) validates the physical VARIANT annotation against the requested Spark type.
- Comet's [scan schema check](https://github.com/apache/datafusion-comet/blob/7e1984399eb887cd13109698ee55cf2ce150f849/spark/src/main/scala/org/apache/comet/rules/CometScanRule.scala#L1067-L1116) examines the requested schema. The manually supplied ordinary struct does not carry the per-field Variant metadata used by `isVariantStruct`, so that fallback does not cover this case.
The failure is confirmed by CI; the routing explanation above is source-derived, without a separately captured execution plan. A fix should validate the file annotation or fall back appropriately, without classifying ordinary structs as Variant solely by their child names.
The other 14 tests in the two restored suites passed. Track this case separately so #5569 can restore that coverage without retaining a whole-suite exclusion. Related: #5438.
Contributor guide
Assessment
This issue has not been assessed yet.