apache / apache/gluten

[VL] Support read when write merge schema

Open
#11,135 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 21h
Merged PRs (30d)
85

Description

### Description

```
[ERROR] TestGlutenDataFrameWriterV2>TestDataFrameWriterV2.testMergeSchemaIgnoreCastingDoubleToFloat:290->SparkTestHelperBase.assertEquals:67->SparkTestHelperBase.assertEquals:87 [Should include new rows with unchanged double column type: row 1 col 1 contents should match]
expected: 1.0
but was: 5.325712093E-315
```
If user ever set TableProperties.SPARK_WRITE_ACCEPT_ANY_SCHEMA and merge schema to true to write the data, source column is double, but the Spark dataframe type is float, iceberg will write the float in parquet file.
```
scala> val df2 = spark.read.parquet("/Users/chengchengjin/code/test/table/data/00000-0-f3c4015d-9e2a-4924-8e42-138b103be3c5-0-00001.parquet")
df2: org.apache.spark.sql.DataFrame = [id: double, data: string]

scala> df2.printSchema()
root
|-- id: double (nullable = true)
|-- data: string (nullable = true)

scala> val df = spark.read.parquet("/Users/chengchengjin/code/test/table/data/00001-6-45ee1a66-11a1-49f4-ad79-5c66c8f67d6d-0-00001.parquet")
df: org.apache.spark.sql.DataFrame = [id: float, data: string]

scala> df.printSchema()
root
|-- id: float (nullable = true)
|-- data: string (nullable = true)

```

This PR https://github.com/apache/iceberg/pull/11419 supports the feature.
Now fallback for table property SPARK_WRITE_ACCEPT_ANY_SCHEMA to true, but user can also alter the table property then offload to native read and with incorrect result.

It's heavy to check the file type and read type in validation.

Velox supports parquet read as other type, but seems not support float as double

### Gluten version

None

Contributor guide

Open the contributing guide

Research direction

Start with TestGlutenDataFrameWriterV2.testMergeSchemaIgnoreCastingDoubleToFloat at line 290 and compare the two parquet schemas shown in the report. Review the SPARK_WRITE_ACCEPT_ANY_SCHEMA and merge-schema behavior, the native-read path, and the referenced PR; done means the reported double/float read case has correct results and a regression test passes.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.