[VL] Exception in GlutenParquetV1QuerySuite decimal read schema evolusion
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 80
Description
### Backend
VL (Velox)
### Bug description
Read decimal(2, 1) as decimal(3, 2)
```
10:51:35.289 ERROR org.apache.spark.task.TaskResources: Task 643 failed by error:
org.apache.gluten.exception.GlutenException: Exception: VeloxRuntimeError
Error Source: RUNTIME
Error Code: INVALID_STATE
Reason: Converted type DECIMAL(2, 1) is not allowed for requested type DECIMAL(3, 2)
Retriable: False
Expression: isCompatible( requestedType, isRepeated, [&](const TypePtr& type) { return type->isShortDecimal() && type->asShortDecimal().precision() >= schemaElementPrecision && type->asShortDecimal().scale() == schemaElementScale; })
Context: Split [Hive: file:///private/var/folders/63/845y6pk53dx_83hpw8ztdchw0000gn/T/spark-d7f8cf57-bac8-4ed7-a1b3-0f5fef463077/part-00000-21ecad37-29b1-49f0-8028-a6f32211c9e3.snappy.parquet 0 - 1055] Task Gluten_Stage_401_TID_643_VTID_401
Additional Context: Operator: TableScan[0] 0
10:51:35.291 ERROR org.apache.spark.executor.Executor: Exception in task 0.0 in stage 401.0 (TID 643)
org.apache.gluten.exception.GlutenException: Exception: VeloxRuntimeError
Error Source: RUNTIME
Error Code: INVALID_STATE
Reason: Converted type DECIMAL(2, 1) is not allowed for requested type DECIMAL(3, 2)
Retriable: False
Expression: isCompatible( requestedType, isRepeated, [&](const TypePtr& type) { return type->isShortDecimal() && type->asShortDecimal().precision() >= schemaElementPrecision && type->asShortDecimal().scale() == schemaElementScale; })
Context: Split [Hive: file:///private/var/folders/63/845y6pk53dx_83hpw8ztdchw0000gn/T/spark-d7f8cf57-bac8-4ed7-a1b3-0f5fef463077/part-00000-21ecad37-29b1-49f0-8028-a6f32211c9e3.snappy.parquet 0 - 1055] Task Gluten_Stage_401_TID_643_VTID_401
Additional Context: Operator: TableScan[0] 0
```
### Gluten version
_No response_
### Spark version
None
### Spark configurations
_No response_
### System information
_No response_
### Relevant logs
```bash
```
Contributor guide
Research direction
Start by locating GlutenParquetV1QuerySuite and the decimal schema-evolution case described in the issue. Reproduce reading decimal(2, 1) as decimal(3, 2) with the Velox backend and inspect the resulting INVALID_STATE error. Done means the read completes without the reported schema compatibility exception.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala
- Domain
- databases, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100