apache / apache/paimon

[Bug] aggregate function in partial update inconsistency

Open
#5,179 12 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.

### Paimon version

1.0.0

### Compute Engine

Flink

### Minimal reproduce step

CREATE TABLE t1
(
k INT,
a INT,
b INT,
g_1 INT,
PRIMARY KEY (k) NOT ENFORCED
) WITH (
'merge-engine' = 'partial-update',
'fields.a.aggregate-function' = 'sum',
'fields.b.aggregate-function' = 'last_non_null_value',
'fields.g_1.sequence-group' = 'a,b'
);

INSERT INTO t1 VALUES (1, 1, 1, 1);
INSERT INTO t1 VALUES (1, 2, 2, 2);
INSERT INTO t1 VALUES (1, 3, 3, 1);

### What doesn't meet your expectations?

when g_1 of third record is smaller, third record should be ignored。
I expect the result to be: 1,3,2,2

### Anything else?

_No response_

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the SQL reproduction for table t1 using partial-update, the sum and last_non_null_value aggregate functions, and sequence group a,b; run it with Paimon 1.0.0 on Flink and compare the third update with the expected result 1,3,2,2. Trace the partial-update handling to determine why a lower g_1 does not cause that record to be ignored, then add regression coverage for this case.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases, stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.