[SQL] Row-based Parquet reader silently accepts incompatible primitive type conversions
- Dominant language
- Scala
- Stars
- 44k
- Forks
- 29.4k
- PR merge metrics
- No merged PRs in 30d
Description
### Problem
Spark's row-based Parquet reader and vectorized Parquet reader handle some incompatible requested
schemas differently. The vectorized reader raises `SchemaColumnConvertNotSupportedException`, while
the row-based reader silently interprets the physical value as the requested Catalyst type.
Two examples are:
| Parquet file type | Requested Spark type | Row-based reader | Vectorized reader |
| --- | --- | --- | --- |
| `FIXED_LEN_BYTE_ARRAY(4)` | `STRING` | Returns the raw bytes as UTF-8 | Rejects the conversion |
| `INT32 (DATE)` | `DECIMAL(10, 0)` | Returns the day count as a decimal | Rejects the conversion |
The row reader's `ParquetRowConverter` currently accepts every binary-like primitive as a string
and treats any `INT32` or `INT64` without decimal metadata as an unannotated integer-backed
decimal. The latter ignores semantic logical annotations such as `DATE`.
This is a correctness problem because changing `spark.sql.parquet.enableVectorizedReader` can
change a query from failing cleanly to returning incorrectly interpreted data.
### Expected behavior
Both readers should reject these unsupported conversions with
`FAILED_READ_FILE.PARQUET_COLUMN_DATA_TYPE_MISMATCH`.
Supported conversions should remain unchanged, including Parquet `BINARY` to Spark `STRING` and
unannotated or signed-integer `INT32`/`INT64` to a sufficiently compatible Spark decimal type.
### Related discussions
* #31319 / SPARK-34212 established the decimal conversion rules and discussed consistency between
the row-based and vectorized readers.
* #44368 / SPARK-40876 added lossless widening conversions to both readers and required logical
annotation checks for semantic types such as `DATE`.
* #29600 / SPARK-32317 proposed a broader configurable conversion matrix but was not merged.
Contributor guide
Research direction
Start by tracing ParquetRowConverter and comparing the row-based and vectorized Parquet readers, focusing on primitive types and logical annotations. Verify that incompatible conversions produce FAILED_READ_FILE.PARQUET_COLUMN_DATA_TYPE_MISMATCH while supported BINARY-to-STRING and compatible integer-to-decimal conversions remain unchanged.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- sql
- Domain
- data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100