apache / apache/gluten

[CH] Gluten Runtime exception when testing decimal overflow

Open
#10,085 0 comments 0 reactions 0 assignees View on GitHub
bug triage
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

### Backend

CH (ClickHouse)

### Bug description

Gluten Runtime exception when testing a decimal overflow scenario.
I am testing and comparing the behavior of vanilla spark and spark with gluten (CH as native engine).
Below are some of the steps to test:

1. create a table
create table tbl_decimal_over(id decimal(38,38)) using parquet.
2. Insert sample test data to the table
insert into tbl_decimal_over values
(0.12345678900987654321123456789009876543),
(0.95400000000000000000000000000000000000)
3. select sum(id) from tbl_decimal_over;

**[Expected behavior]**
**vanilla spark -** should return null in the above case.
**gluten with CH as native -** should return null

**[actual behavior]**
**vanilla spark -** null (as expected).
**gluten with CH as native -** results in RuntimeException
java.lang.RuntimeException: Error while decoding: java.lang.Arithmetic: Decimal precision 39 exceeds max precision 38.

### Gluten version

main branch

### Spark version

Spark-3.3.x

### Spark configurations

_No response_

### System information

_No response_

### Relevant logs

```bash
java.lang.RuntimeException: Error while decoding: java.lang.Arithmetic: Decimal precision 39 exceeds max precision 38.
createexternalrow(input[0, decimal(38,38), true].toJavaBigDecimal, StructField(sum(id), DecimalType(38,38), true))
at org.apache.spark.sql.errors.QueryExecutionErrors$.expressionDecodingError
```

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the decimal overflow with the SQL steps in this issue on Spark 3.3.x with the ClickHouse native engine, then trace the reported expressionDecodingError and Decimal precision 39 message. Done means the query returns null, matching vanilla Spark, without a RuntimeException.

Written by the indexing model from the issue text.

Assessment

Tech stack
clickhouse, scala, spark
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.