[VL] Result mismatch from test case: "SPARK-37196: HiveDecimal Precision Scale match failed should return null"
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 80
Description
### Backend
VL (Velox)
### Bug description
We noticed result mismatch from above test case. null should be returned if table schema doesn't match the schema from files. but below error is thrown.
```
Error Source: RUNTIME
Error Code: INVALID_STATE
Reason: Unsupported value size: 16
Retriable: False
DisplayReason: False
Context: Split [Hive: XXX.snappy.parquet 0 - 798] Task Gluten_Stage_4_TID_4_VTID_4
Additional Context: Operator: TableScan[0] 0
Function: getIntValues
File: /var/git/Velox/velox/dwio/common/SelectiveColumnReader.cpp
Line: 216
Stack trace:
# 0 _ZN8facebook5velox7process10StackTraceC1Ei
# 1 _ZN8facebook5velox14VeloxExceptionC2EPKcmS3_St17basic_string_viewIcSt11char_traitsIcEES7_S7_S7_bbNS1_4TypeES7_
# 2 _ZN8facebook5velox6detail14veloxCheckFailINS0_17VeloxRuntimeErrorERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEEvRKNS1_18VeloxCheckFailArgsET0_
# 3 _ZN8facebook5velox4dwio6common21SelectiveColumnReader12getIntValuesEN5folly5RangeIPKiEERKSt10shared_ptrIKNS0_4TypeEEPS9_INS0_10BaseVectorEE
# 4 _ZN8facebook5velox7parquet19IntegerColumnReader9getValuesEN5folly5RangeIPKiEEPSt10shared_ptrINS0_10BaseVectorEE
# 5 _ZN8facebook5velox4dwio6common31SelectiveStructColumnReaderBase9getValuesEN5folly5RangeIPKiEEPSt10shared_ptrINS0_10BaseVectorEE
# 6 _ZN8facebook5velox4dwio6common31SelectiveStructColumnReaderBase4nextEmRSt10shared_ptrINS0_10BaseVectorEEPKNS2_8MutationE
# 7 _ZN8facebook5velox7parquet16ParquetRowReader4nextEmRSt10shared_ptrINS0_10BaseVectorEEPKNS0_4dwio6common8MutationE
# 8 _ZN8facebook5velox9connector4hive14HiveDataSource4nextEmRN5folly10SemiFutureINS4_4UnitEEE
# 9 _ZN8facebook5velox4exec9TableScan9getOutputEv
# 10 _ZN8facebook5velox4exec6Driver11runInternalERSt10shared_ptrIS2_ERS3_INS1_13BlockingStateEERS3_INS0_9RowVectorEE
# 11 _ZN8facebook5velox4exec6Driver4nextERSt10shared_ptrINS1_13BlockingStateEE
# 12 _ZN8facebook5velox4exec4Task4nextEPN5folly10SemiFutureINS3_4UnitEEE
# 13 _ZN6gluten24WholeStageResultIterator4nextEv
# 14 Java_org_apache_gluten_vectorized_ColumnarBatchOutIterator_nativeHasNext
# 15 0x00007fe9087dda10
```
cc @zhouyuan @rui-mo @PHILO-HE
Looks error should be thrown only when `spark.sql.hive.convertMetastoreParquet` is true. Should we fallback the hive scan if `spark.sql.hive.convertMetastoreParquet` is false?
### Spark version
None
### Spark configurations
_No response_
### System information
_No response_
### Relevant logs
```bash
```
Contributor guide
Research direction
Start with the reported Velox path in dwio/common/SelectiveColumnReader.cpp, especially getIntValues, and trace the Parquet IntegerColumnReader through TableScan. Reproduce the SPARK-37196 HiveDecimal precision/scale mismatch and inspect the spark.sql.hive.convertMetastoreParquet setting. Done means the mismatch returns null or follows the appropriate Hive-scan fallback instead of raising INVALID_STATE.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, scala
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100