apache / apache/paimon

[Bug] Decimal Value Corruption in Batch Write (Values Shifted Incorrectly)

Open
#5,656 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.

### Paimon version

From paimon-bundle version 0.9.0 to 1.1.1

### Compute Engine

flink 1.19.1

### Minimal reproduce step

Description:

I'm using paimon-bundle for Batch Write operations and encountering incorrect decimal field values after writing. The issue persists across versions from 0.9.0 to 1.1.1.

Observed Behavior:
1、When writing a 2-decimal-place value (e.g., 1.23), the result gets left-shifted by 2 places (becomes 0.0123)
2、When writing a 3-decimal-place value, the result gets left-shifted by 1 place
3、4-decimal-place values appear correct

Reproduction Code:
SQL:
```
CREATE CATALOG xxxx ....'type' = 'paimon'....;
use xxxx;
CREATE TABLE `tmp`.`test` (
`aa` BIGINT NOT NULL ,
`bb` TIMESTAMP ( 3 ),
`aaa` STRING ,
`bbb` DECIMAL ( 12, 4 ) ,
CONSTRAINT `PK_aa` PRIMARY KEY ( `aa` ) NOT ENFORCED
) WITH ( )
```
JAVA:
```
Identifier identifier = Identifier.create(dbName, tableName);
org.apache.paimon.table.Table table = catalog.getTable(identifier);
BatchWriteBuilder writeBuilder = table.newBatchWriteBuilder();
BatchTableWrite write = writeBuilder.newWrite();

// First record
Long r1_col1 = Long.valueOf(111);
BinaryString r1_col2 = BinaryString.fromString("aaa");
BigDecimal bd = new BigDecimal("1.23");
Decimal r1_col3 = org.apache.paimon.data.Decimal.fromBigDecimal(bd, bd.precision(), bd.scale());
System.out.println(r1_col3);
GenericRow record1 = GenericRow.of(r1_col1, null, r1_col2, r1_col3);
write.write(record1, 1); // Using fixed bucket number 1

// Second record
Long r2_col1 = Long.valueOf(222);
BinaryString r2_col2 = BinaryString.fromString("bbb");
BigDecimal bd2 = new BigDecimal("333333.3333");
Decimal r2_col3 = org.apache.paimon.data.Decimal.fromBigDecimal(bd2, bd2.precision(), bd2.scale());
GenericRow record2 = GenericRow.of(r2_col1, null, r2_col2, r2_col3);
write.write(record2, 1);

List messages = write.prepareCommit();
BatchTableCommit commit = writeBuilder.newCommit();
commit.commit(messages);
System.out.println("done");
```

Actual Output:
0.0123
333333.3333

### What doesn't meet your expectations?

The correct query output should be:
1.23
333333.3333

### Anything else?

_No response_

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by running the supplied SQL and Java reproduction with paimon-bundle 0.9.0–1.1.1, focusing on Decimal.fromBigDecimal and the BatchWriteBuilder/BatchTableWrite path. Compare the stored query results for 1.23 and 333333.3333 with the expected values; done means decimal values retain their declared scale during batch writes.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, sql
Domain
data-engineering, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.