[VL] Support read when write merge schema
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 85
Description
### Description
```
[ERROR] TestGlutenDataFrameWriterV2>TestDataFrameWriterV2.testMergeSchemaIgnoreCastingDoubleToFloat:290->SparkTestHelperBase.assertEquals:67->SparkTestHelperBase.assertEquals:87 [Should include new rows with unchanged double column type: row 1 col 1 contents should match]
expected: 1.0
but was: 5.325712093E-315
```
If user ever set TableProperties.SPARK_WRITE_ACCEPT_ANY_SCHEMA and merge schema to true to write the data, source column is double, but the Spark dataframe type is float, iceberg will write the float in parquet file.
```
scala> val df2 = spark.read.parquet("/Users/chengchengjin/code/test/table/data/00000-0-f3c4015d-9e2a-4924-8e42-138b103be3c5-0-00001.parquet")
df2: org.apache.spark.sql.DataFrame = [id: double, data: string]
scala> df2.printSchema()
root
|-- id: double (nullable = true)
|-- data: string (nullable = true)
scala> val df = spark.read.parquet("/Users/chengchengjin/code/test/table/data/00001-6-45ee1a66-11a1-49f4-ad79-5c66c8f67d6d-0-00001.parquet")
df: org.apache.spark.sql.DataFrame = [id: float, data: string]
scala> df.printSchema()
root
|-- id: float (nullable = true)
|-- data: string (nullable = true)
```
This PR https://github.com/apache/iceberg/pull/11419 supports the feature.
Now fallback for table property SPARK_WRITE_ACCEPT_ANY_SCHEMA to true, but user can also alter the table property then offload to native read and with incorrect result.
It's heavy to check the file type and read type in validation.
Velox supports parquet read as other type, but seems not support float as double
### Gluten version
None
Contributor guide
Research direction
Start with TestGlutenDataFrameWriterV2.testMergeSchemaIgnoreCastingDoubleToFloat at line 290 and compare the two parquet schemas shown in the report. Review the SPARK_WRITE_ACCEPT_ANY_SCHEMA and merge-schema behavior, the native-read path, and the referenced PR; done means the reported double/float read case has correct results and a regression test passes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100