[Bug] aggregate function in partial update inconsistency
- Dominant language
- Java
- Stars
- 3.4k
- Forks
- 1.4k
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 396
Description
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.
### Paimon version
1.0.0
### Compute Engine
Flink
### Minimal reproduce step
CREATE TABLE t1
(
k INT,
a INT,
b INT,
g_1 INT,
PRIMARY KEY (k) NOT ENFORCED
) WITH (
'merge-engine' = 'partial-update',
'fields.a.aggregate-function' = 'sum',
'fields.b.aggregate-function' = 'last_non_null_value',
'fields.g_1.sequence-group' = 'a,b'
);
INSERT INTO t1 VALUES (1, 1, 1, 1);
INSERT INTO t1 VALUES (1, 2, 2, 2);
INSERT INTO t1 VALUES (1, 3, 3, 1);
### What doesn't meet your expectations?
when g_1 of third record is smaller, third record should be ignored。
I expect the result to be: 1,3,2,2
### Anything else?
_No response_
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the SQL reproduction for table t1 using partial-update, the sum and last_non_null_value aggregate functions, and sequence group a,b; run it with Paimon 1.0.0 on Flink and compare the third update with the expected result 1,3,2,2. Trace the partial-update handling to determine why a lower g_1 does not cause that record to be ignored, then add regression coverage for this case.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases, stream-processing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100