apache / apache/paimon

[Bug] first_value semantic conflict

Open
#3,393 6 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.

### Paimon version

0.8

### Compute Engine

Flink

### Minimal reproduce step

I have read this issues: https://github.com/apache/paimon/issues/3020

I discovered this phenomenon:
```
CREATE TABLE T (
k INT,
a INT,
b VARCHAR,
c VARCHAR,
d VARCHAR,
PRIMARY KEY (k) NOT ENFORCED)
WITH ('merge-engine'='aggregation',
'fields.b.aggregate-function'='first_value',
'fields.c.aggregate-function'='first_non_null_value',
'fields.d.aggregate-function'='first_not_null_value',
'sequence.field' = 'a'
);
```
```
INSERT INTO T VALUES (1, 4, '4', '4', '4')
INSERT INTO T VALUES (1, 2, '2', '2', '2')
INSERT INTO T VALUES (1, 3, '3', '3', '3')
```
The result is `[1, 4, 2, 2, 2]` , as expected

If I manually trigger a compaction before the third insert, like:
```
INSERT INTO T VALUES (1, 4, '4', '4', '4')
INSERT INTO T VALUES (1, 2, '2', '2', '2')
CALL sys.compact('default.t', '', '', '', 'sink.parallelism=4')
INSERT INTO T VALUES (1, 3, '3', '3', '3')
```
The result is `[1, 4, 3, 3, 3]` , and there is a semantic problem.
For local debugging, in this case, there are only two values merged ​​([1, 3, 3, 3, 3], [1, 4, 2, 2, 2]) when querying.
It seems that after compaction, the previous sequence information was lost.

### What doesn't meet your expectations?

The Doc. description is vague:

![1716547483757](https://github.com/apache/paimon/assets/72678624/8dafd977-2139-4d04-b128-53e21dffd552)

I'm not sure if there's something wrong with my understanding. Should the oldest sequence be maintained? Otherwise the compaction will lead to inconsistent first_value semantics

### Anything else?

_No response_

### Are you willing to submit a PR?

- [X] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the SQL sequence with aggregation fields using first_value, first_non_null_value, and first_not_null_value, then repeat it with CALL sys.compact before the third insert. Compare the results and the retained sequence information before and after compaction; the issue is resolved when compaction preserves consistent first_value semantics or the documented behavior is corrected.

Written by the indexing model from the issue text.

Assessment

Tech stack
sql
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.