apache / apache/gluten

[VL] Exception in GlutenParquetV1QuerySuite decimal read schema evolusion

Open
#11,219 0 comments 0 reactions 0 assignees View on GitHub
bug triage
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

### Backend

VL (Velox)

### Bug description

Read decimal(2, 1) as decimal(3, 2)
```
10:51:35.289 ERROR org.apache.spark.task.TaskResources: Task 643 failed by error:
org.apache.gluten.exception.GlutenException: Exception: VeloxRuntimeError
Error Source: RUNTIME
Error Code: INVALID_STATE
Reason: Converted type DECIMAL(2, 1) is not allowed for requested type DECIMAL(3, 2)
Retriable: False
Expression: isCompatible( requestedType, isRepeated, [&](const TypePtr& type) { return type->isShortDecimal() && type->asShortDecimal().precision() >= schemaElementPrecision && type->asShortDecimal().scale() == schemaElementScale; })
Context: Split [Hive: file:///private/var/folders/63/845y6pk53dx_83hpw8ztdchw0000gn/T/spark-d7f8cf57-bac8-4ed7-a1b3-0f5fef463077/part-00000-21ecad37-29b1-49f0-8028-a6f32211c9e3.snappy.parquet 0 - 1055] Task Gluten_Stage_401_TID_643_VTID_401
Additional Context: Operator: TableScan[0] 0
10:51:35.291 ERROR org.apache.spark.executor.Executor: Exception in task 0.0 in stage 401.0 (TID 643)
org.apache.gluten.exception.GlutenException: Exception: VeloxRuntimeError
Error Source: RUNTIME
Error Code: INVALID_STATE
Reason: Converted type DECIMAL(2, 1) is not allowed for requested type DECIMAL(3, 2)
Retriable: False
Expression: isCompatible( requestedType, isRepeated, [&](const TypePtr& type) { return type->isShortDecimal() && type->asShortDecimal().precision() >= schemaElementPrecision && type->asShortDecimal().scale() == schemaElementScale; })
Context: Split [Hive: file:///private/var/folders/63/845y6pk53dx_83hpw8ztdchw0000gn/T/spark-d7f8cf57-bac8-4ed7-a1b3-0f5fef463077/part-00000-21ecad37-29b1-49f0-8028-a6f32211c9e3.snappy.parquet 0 - 1055] Task Gluten_Stage_401_TID_643_VTID_401
Additional Context: Operator: TableScan[0] 0
```

### Gluten version

_No response_

### Spark version

None

### Spark configurations

_No response_

### System information

_No response_

### Relevant logs

```bash

```

Contributor guide

Open the contributing guide

Research direction

Start by locating GlutenParquetV1QuerySuite and the decimal schema-evolution case described in the issue. Reproduce reading decimal(2, 1) as decimal(3, 2) with the Velox backend and inspect the resulting INVALID_STATE error. Done means the read completes without the reported schema compatibility exception.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala
Domain
databases, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.