apache / apache/gluten

[VL] Result mismatch from test case: "SPARK-37196: HiveDecimal Precision Scale match failed should return null"

Open
#8,729 2 comments 1 reaction 0 assignees View on GitHub
bug triage
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

### Backend

VL (Velox)

### Bug description

We noticed result mismatch from above test case. null should be returned if table schema doesn't match the schema from files. but below error is thrown.
```
Error Source: RUNTIME
Error Code: INVALID_STATE
Reason: Unsupported value size: 16
Retriable: False
DisplayReason: False
Context: Split [Hive: XXX.snappy.parquet 0 - 798] Task Gluten_Stage_4_TID_4_VTID_4
Additional Context: Operator: TableScan[0] 0
Function: getIntValues
File: /var/git/Velox/velox/dwio/common/SelectiveColumnReader.cpp
Line: 216
Stack trace:
# 0 _ZN8facebook5velox7process10StackTraceC1Ei
# 1 _ZN8facebook5velox14VeloxExceptionC2EPKcmS3_St17basic_string_viewIcSt11char_traitsIcEES7_S7_S7_bbNS1_4TypeES7_
# 2 _ZN8facebook5velox6detail14veloxCheckFailINS0_17VeloxRuntimeErrorERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEEvRKNS1_18VeloxCheckFailArgsET0_
# 3 _ZN8facebook5velox4dwio6common21SelectiveColumnReader12getIntValuesEN5folly5RangeIPKiEERKSt10shared_ptrIKNS0_4TypeEEPS9_INS0_10BaseVectorEE
# 4 _ZN8facebook5velox7parquet19IntegerColumnReader9getValuesEN5folly5RangeIPKiEEPSt10shared_ptrINS0_10BaseVectorEE
# 5 _ZN8facebook5velox4dwio6common31SelectiveStructColumnReaderBase9getValuesEN5folly5RangeIPKiEEPSt10shared_ptrINS0_10BaseVectorEE
# 6 _ZN8facebook5velox4dwio6common31SelectiveStructColumnReaderBase4nextEmRSt10shared_ptrINS0_10BaseVectorEEPKNS2_8MutationE
# 7 _ZN8facebook5velox7parquet16ParquetRowReader4nextEmRSt10shared_ptrINS0_10BaseVectorEEPKNS0_4dwio6common8MutationE
# 8 _ZN8facebook5velox9connector4hive14HiveDataSource4nextEmRN5folly10SemiFutureINS4_4UnitEEE
# 9 _ZN8facebook5velox4exec9TableScan9getOutputEv
# 10 _ZN8facebook5velox4exec6Driver11runInternalERSt10shared_ptrIS2_ERS3_INS1_13BlockingStateEERS3_INS0_9RowVectorEE
# 11 _ZN8facebook5velox4exec6Driver4nextERSt10shared_ptrINS1_13BlockingStateEE
# 12 _ZN8facebook5velox4exec4Task4nextEPN5folly10SemiFutureINS3_4UnitEEE
# 13 _ZN6gluten24WholeStageResultIterator4nextEv
# 14 Java_org_apache_gluten_vectorized_ColumnarBatchOutIterator_nativeHasNext
# 15 0x00007fe9087dda10
```
cc @zhouyuan @rui-mo @PHILO-HE
Looks error should be thrown only when `spark.sql.hive.convertMetastoreParquet` is true. Should we fallback the hive scan if `spark.sql.hive.convertMetastoreParquet` is false?

### Spark version

None

### Spark configurations

_No response_

### System information

_No response_

### Relevant logs

```bash

```

Contributor guide

Open the contributing guide

Research direction

Start with the reported Velox path in dwio/common/SelectiveColumnReader.cpp, especially getIntValues, and trace the Parquet IntegerColumnReader through TableScan. Reproduce the SPARK-37196 HiveDecimal precision/scale mismatch and inspect the spark.sql.hive.convertMetastoreParquet setting. Done means the mismatch returns null or follows the appropriate Hive-scan fallback instead of raising INVALID_STATE.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, scala
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.