apache / apache/datafusion-comet

Spark 4.1 NullType parquet: parquet-rs rejects BOOLEAN + Unknown logical type

Open
#4,199 2 comments 0 reactions 0 assignees View on GitHub
area:scan bug priority:medium spark 4.1
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 4h
Merged PRs (30d)
198

Description

## Description

Spark writes \`NullType\` columns to parquet as \`BOOLEAN\` physical type with an \`Unknown\` logical type annotation (comment in [\`ParquetSchemaConverter.scala\`](https://github.com/apache/spark/blob/v4.1.1/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetSchemaConverter.scala#L877): _"Selected primitive type here doesn't have significance"_). parquet-rs only accepts \`LogicalType::Unknown\` paired with \`PhysicalType::INT32\` and rejects any other physical type with \`Cannot annotate Unknown from BOOLEAN for field '…'\` ([parquet-57.2.0/src/schema/types.rs:401, :423](https://docs.rs/parquet/57.2.0/parquet/schema/types/index.html)).

Result: any attempt to read a Spark-written parquet file that contains a \`NullType\` field fails in Comet with:

\`\`\`
org.apache.comet.CometNativeException: Parquet error: Cannot annotate Unknown from BOOLEAN for field '_1'
\`\`\`

The SPARK-54220 test in \`ParquetIOSuite\` (\`SPARK-54220: vectorized reader: missing all struct fields, struct with NullType only\`) is the concrete reproducer. It was unignored as part of PR #4190 / issue #4136 but crashes on the parquet read path before the new fix in \`parquet_convert_struct_to_struct\` is reached.

## Reproducer

See \`issue #4136: struct with only NullType fields in file (SPARK-54220)\` in \`CometNativeReaderSuite\`. The failure manifests for both \`native_datafusion\` and \`native_iceberg_compat\`.

## Suspected fix

Either:

1. Upstream parquet-rs to accept \`(Unknown, BOOLEAN)\` (and arguably any physical type, since Spark's comment makes clear the physical type is a don't-care), or
2. Work around in Comet: in the schema adapter or parquet reader factory, rewrite the physical type to INT32 before passing it to parquet-rs' validator — or special-case the Unknown-annotated field at read time.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.