apache / apache/datafusion-comet
Spark 4.1 NullType parquet: parquet-rs rejects BOOLEAN + Unknown logical type
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 198
Description
## Description
Spark writes \`NullType\` columns to parquet as \`BOOLEAN\` physical type with an \`Unknown\` logical type annotation (comment in [\`ParquetSchemaConverter.scala\`](https://github.com/apache/spark/blob/v4.1.1/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetSchemaConverter.scala#L877): _"Selected primitive type here doesn't have significance"_). parquet-rs only accepts \`LogicalType::Unknown\` paired with \`PhysicalType::INT32\` and rejects any other physical type with \`Cannot annotate Unknown from BOOLEAN for field '…'\` ([parquet-57.2.0/src/schema/types.rs:401, :423](https://docs.rs/parquet/57.2.0/parquet/schema/types/index.html)).
Result: any attempt to read a Spark-written parquet file that contains a \`NullType\` field fails in Comet with:
\`\`\`
org.apache.comet.CometNativeException: Parquet error: Cannot annotate Unknown from BOOLEAN for field '_1'
\`\`\`
The SPARK-54220 test in \`ParquetIOSuite\` (\`SPARK-54220: vectorized reader: missing all struct fields, struct with NullType only\`) is the concrete reproducer. It was unignored as part of PR #4190 / issue #4136 but crashes on the parquet read path before the new fix in \`parquet_convert_struct_to_struct\` is reached.
## Reproducer
See \`issue #4136: struct with only NullType fields in file (SPARK-54220)\` in \`CometNativeReaderSuite\`. The failure manifests for both \`native_datafusion\` and \`native_iceberg_compat\`.
## Suspected fix
Either:
1. Upstream parquet-rs to accept \`(Unknown, BOOLEAN)\` (and arguably any physical type, since Spark's comment makes clear the physical type is a don't-care), or
2. Work around in Comet: in the schema adapter or parquet reader factory, rewrite the physical type to INT32 before passing it to parquet-rs' validator — or special-case the Unknown-annotated field at read time.
Contributor guide
Assessment
This issue has not been assessed yet.