apache / apache/spark

[SQL] Row-based Parquet reader silently accepts incompatible primitive type conversions

Open
#58,495 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Scala
Stars
44k
Forks
29.4k
PR merge metrics
No merged PRs in 30d

Description

### Problem

Spark's row-based Parquet reader and vectorized Parquet reader handle some incompatible requested
schemas differently. The vectorized reader raises `SchemaColumnConvertNotSupportedException`, while
the row-based reader silently interprets the physical value as the requested Catalyst type.

Two examples are:

| Parquet file type | Requested Spark type | Row-based reader | Vectorized reader |
| --- | --- | --- | --- |
| `FIXED_LEN_BYTE_ARRAY(4)` | `STRING` | Returns the raw bytes as UTF-8 | Rejects the conversion |
| `INT32 (DATE)` | `DECIMAL(10, 0)` | Returns the day count as a decimal | Rejects the conversion |

The row reader's `ParquetRowConverter` currently accepts every binary-like primitive as a string
and treats any `INT32` or `INT64` without decimal metadata as an unannotated integer-backed
decimal. The latter ignores semantic logical annotations such as `DATE`.

This is a correctness problem because changing `spark.sql.parquet.enableVectorizedReader` can
change a query from failing cleanly to returning incorrectly interpreted data.

### Expected behavior

Both readers should reject these unsupported conversions with
`FAILED_READ_FILE.PARQUET_COLUMN_DATA_TYPE_MISMATCH`.

Supported conversions should remain unchanged, including Parquet `BINARY` to Spark `STRING` and
unannotated or signed-integer `INT32`/`INT64` to a sufficiently compatible Spark decimal type.

### Related discussions

* #31319 / SPARK-34212 established the decimal conversion rules and discussed consistency between
the row-based and vectorized readers.
* #44368 / SPARK-40876 added lossless widening conversions to both readers and required logical
annotation checks for semantic types such as `DATE`.
* #29600 / SPARK-32317 proposed a broader configurable conversion matrix but was not merged.

Contributor guide

Open the contributing guide

Research direction

Start by tracing ParquetRowConverter and comparing the row-based and vectorized Parquet readers, focusing on primitive types and logical annotations. Verify that incompatible conversions produce FAILED_READ_FILE.PARQUET_COLUMN_DATA_TYPE_MISMATCH while supported BINARY-to-STRING and compatible integer-to-decimal conversions remain unchanged.

Written by the indexing model from the issue text.

Assessment

Tech stack
sql
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.