apache / apache/datafusion

decimal calculate overflow but not throw error

Open
#16,406 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Describe the bug

1. make test csv
```
import csv
import random
import decimal

random.seed(42)

def make_big_random_decimal():
n = random.randint(1, 1 << 53)
p = pow(10, random.randint(1, 8))
return decimal.Decimal(n) / decimal.Decimal(p)

def make_small_random_decimal():
n = random.randint(1, 100)
return decimal.Decimal(f"0.{n}")

decimals_1 = [make_big_random_decimal() for _ in range(100000)]
decimals_2 = [make_small_random_decimal() for _ in range(100000)]

SUM = decimal.Decimal(0)

with open("/tmp/decimal.csv", 'w') as f:
f = csv.writer(f)
for d1, d2 in zip(decimals_1, decimals_2):
f.writerow([d1, d2])
SUM += d1 * d2

print(SUM) // print 3318680488765741748.466457758
```

2. calculate sum(d1*d2) in datafusion
```
use arrow_schema::{DataType, Field, Schema, SchemaBuilder};
use datafusion::error::Result;
use datafusion::prelude::*;

#[tokio::main]
async fn main() -> Result<()> {
let ctx = SessionContext::new();
let schema = Schema::new(vec![
Field::new("d1", DataType::Decimal128(38, 10), false),
Field::new("d2", DataType::Decimal128(38, 10), false),
]);
let schema = SchemaBuilder::from(schema.fields).finish();
let options = CsvReadOptions::new()
.schema(&schema)
.has_header(false)
.file_extension(".csv");
ctx.register_csv("tb", "/tmp/decimal.csv", options).await?;
ctx.sql("select sum(d1 * d2) from tb").await?.show().await?;
Ok(())
}

+-----------------------------------------+
| sum(tb.d1 * tb.d2) |
+-----------------------------------------+
| -84143180443642886.16728833341768211456 |
+-----------------------------------------+
```

### To Reproduce

_No response_

### Expected behavior

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

No repository file or test is named. Start by running the provided Rust DataFusion reproduction against the generated CSV and tracing decimal multiplication and SUM handling; done means the overflow behavior is explicitly decided and covered by a regression test without silently producing the incorrect result.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, rust
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.