[CH] Gluten Runtime exception when testing decimal overflow
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 80
Description
### Backend
CH (ClickHouse)
### Bug description
Gluten Runtime exception when testing a decimal overflow scenario.
I am testing and comparing the behavior of vanilla spark and spark with gluten (CH as native engine).
Below are some of the steps to test:
1. create a table
create table tbl_decimal_over(id decimal(38,38)) using parquet.
2. Insert sample test data to the table
insert into tbl_decimal_over values
(0.12345678900987654321123456789009876543),
(0.95400000000000000000000000000000000000)
3. select sum(id) from tbl_decimal_over;
**[Expected behavior]**
**vanilla spark -** should return null in the above case.
**gluten with CH as native -** should return null
**[actual behavior]**
**vanilla spark -** null (as expected).
**gluten with CH as native -** results in RuntimeException
java.lang.RuntimeException: Error while decoding: java.lang.Arithmetic: Decimal precision 39 exceeds max precision 38.
### Gluten version
main branch
### Spark version
Spark-3.3.x
### Spark configurations
_No response_
### System information
_No response_
### Relevant logs
```bash
java.lang.RuntimeException: Error while decoding: java.lang.Arithmetic: Decimal precision 39 exceeds max precision 38.
createexternalrow(input[0, decimal(38,38), true].toJavaBigDecimal, StructField(sum(id), DecimalType(38,38), true))
at org.apache.spark.sql.errors.QueryExecutionErrors$.expressionDecodingError
```
Contributor guide
Research direction
Start by reproducing the decimal overflow with the SQL steps in this issue on Spark 3.3.x with the ClickHouse native engine, then trace the reported expressionDecodingError and Decimal precision 39 message. Done means the query returns null, matching vanilla Spark, without a RuntimeException.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- clickhouse, scala, spark
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 28/100