decimal calculate overflow but not throw error
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Describe the bug
1. make test csv
```
import csv
import random
import decimal
random.seed(42)
def make_big_random_decimal():
n = random.randint(1, 1 << 53)
p = pow(10, random.randint(1, 8))
return decimal.Decimal(n) / decimal.Decimal(p)
def make_small_random_decimal():
n = random.randint(1, 100)
return decimal.Decimal(f"0.{n}")
decimals_1 = [make_big_random_decimal() for _ in range(100000)]
decimals_2 = [make_small_random_decimal() for _ in range(100000)]
SUM = decimal.Decimal(0)
with open("/tmp/decimal.csv", 'w') as f:
f = csv.writer(f)
for d1, d2 in zip(decimals_1, decimals_2):
f.writerow([d1, d2])
SUM += d1 * d2
print(SUM) // print 3318680488765741748.466457758
```
2. calculate sum(d1*d2) in datafusion
```
use arrow_schema::{DataType, Field, Schema, SchemaBuilder};
use datafusion::error::Result;
use datafusion::prelude::*;
#[tokio::main]
async fn main() -> Result<()> {
let ctx = SessionContext::new();
let schema = Schema::new(vec![
Field::new("d1", DataType::Decimal128(38, 10), false),
Field::new("d2", DataType::Decimal128(38, 10), false),
]);
let schema = SchemaBuilder::from(schema.fields).finish();
let options = CsvReadOptions::new()
.schema(&schema)
.has_header(false)
.file_extension(".csv");
ctx.register_csv("tb", "/tmp/decimal.csv", options).await?;
ctx.sql("select sum(d1 * d2) from tb").await?.show().await?;
Ok(())
}
+-----------------------------------------+
| sum(tb.d1 * tb.d2) |
+-----------------------------------------+
| -84143180443642886.16728833341768211456 |
+-----------------------------------------+
```
### To Reproduce
_No response_
### Expected behavior
_No response_
### Additional context
_No response_
Contributor guide
Research direction
No repository file or test is named. Start by running the provided Rust DataFusion reproduction against the generated CSV and tracing decimal multiplication and SUM handling; done means the overflow behavior is explicitly decided and covered by a regression test without silently producing the incorrect result.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, rust
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 42/100