[Bug] first_value semantic conflict
- Dominant language
- Java
- Stars
- 3.4k
- Forks
- 1.4k
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 396
Description
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.
### Paimon version
0.8
### Compute Engine
Flink
### Minimal reproduce step
I have read this issues: https://github.com/apache/paimon/issues/3020
I discovered this phenomenon:
```
CREATE TABLE T (
k INT,
a INT,
b VARCHAR,
c VARCHAR,
d VARCHAR,
PRIMARY KEY (k) NOT ENFORCED)
WITH ('merge-engine'='aggregation',
'fields.b.aggregate-function'='first_value',
'fields.c.aggregate-function'='first_non_null_value',
'fields.d.aggregate-function'='first_not_null_value',
'sequence.field' = 'a'
);
```
```
INSERT INTO T VALUES (1, 4, '4', '4', '4')
INSERT INTO T VALUES (1, 2, '2', '2', '2')
INSERT INTO T VALUES (1, 3, '3', '3', '3')
```
The result is `[1, 4, 2, 2, 2]` , as expected
If I manually trigger a compaction before the third insert, like:
```
INSERT INTO T VALUES (1, 4, '4', '4', '4')
INSERT INTO T VALUES (1, 2, '2', '2', '2')
CALL sys.compact('default.t', '', '', '', 'sink.parallelism=4')
INSERT INTO T VALUES (1, 3, '3', '3', '3')
```
The result is `[1, 4, 3, 3, 3]` , and there is a semantic problem.
For local debugging, in this case, there are only two values merged ([1, 3, 3, 3, 3], [1, 4, 2, 2, 2]) when querying.
It seems that after compaction, the previous sequence information was lost.
### What doesn't meet your expectations?
The Doc. description is vague:

I'm not sure if there's something wrong with my understanding. Should the oldest sequence be maintained? Otherwise the compaction will lead to inconsistent first_value semantics
### Anything else?
_No response_
### Are you willing to submit a PR?
- [X] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Reproduce the SQL sequence with aggregation fields using first_value, first_non_null_value, and first_not_null_value, then repeat it with CALL sys.compact before the third insert. Compare the results and the retained sequence information before and after compaction; the issue is resolved when compaction preserves consistent first_value semantics or the documented behavior is corrected.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- sql
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100