apache / apache/paimon

[Bug] AppendOnly table full reading didn't output full data by window TVF

Open
#827 6 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/incubator-paimon/issues) and found nothing similar.

### Paimon version

paimon-flink-1.14-0.4-20230331.003039-13.jar

### Compute Engine

flink-1.14.5

### Minimal reproduce step

I have an appendOnly table with watermark set, and 'scan.mode' = 'latest-full'. I read data from it for window aggregation by window TVF. I expect the result to be able to output complete historical data, but from the actual situation, it seems Only the latest compact snapshot data is read (However, judging from the input and output volume of the operator on the web ui, the full amount of data has been read). For example: the earliest piece of data in the latest snapshot is at 2023-04-04 00:00, and the latest compact is at 2023-04-04 15:35, the expected result is to output the window data of every minute after 2023-04-04 00:00, but the actual output is from 2023-04-04 15:35, I am sure My 'scan.mode' is set correctly. The relevant flink sql is as follows:

```
CREATE TABLE log_wk (
xxx xxx,
xxx xxx,
xxx xxx,
`event_time` as TO_TIMESTAMP_LTZ(xxx,3),
WATERMARK FOR event_time AS event_time - INTERVAL '5' SECOND
) PARTITIONED BY (datekey, event) WITH (
'bucket' = '2',
'bucket-key' = 'xxx',
'scan.mode' = 'latest-full',
'write-mode' = 'append-only',
'sink.parallelism'='2'
);

SELECT window_start, window_end, MAX(xxx) ax max_val
FROM TABLE(
TUMBLE(TABLE log_wk, DESCRIPTOR(event_time), INTERVAL '1' MINUTES)
)
GROUP BY window_start, window_end;
```

the snapshot history information fragment of the table:

| +I | 329 | 14 | 52aef258-6c2f-47bb-8840-0ee... | 298 | APPEND | 2023-04-04 16:25:32.578 | 38784131 | 6697 | 0 | 1680596725739 |
| +I | 330 | 14 | 52aef258-6c2f-47bb-8840-0ee... | 298 | COMPACT | 2023-04-04 16:25:32.682 | 46688935 | 7904804 | 0 | 1680596725739 |
| +I | 331 | 14 | 52aef258-6c2f-47bb-8840-0ee... | 299 | APPEND | 2023-04-04 16:26:32.524 | 46696450 | 7515 | 0 | 1680596786580 |

output result:
+----+-------------------------+-------------------------+----------------------+
| op | window_start | window_end | max_val |
+----+-------------------------+-------------------------+----------------------+
| +I | 2023-04-04 16:24:00.000 | 2023-04-04 16:25:00.000 | 1680596699400 |
| +I | 2023-04-04 16:25:00.000 | 2023-04-04 16:26:00.000 | 1680596759778 |
| +I | 2023-04-04 16:26:00.000 | 2023-04-04 16:27:00.000 | 1680596819993 |
| +I | 2023-04-04 16:27:00.000 | 2023-04-04 16:28:00.000 | 1680596879427 |
| +I | 2023-04-04 16:28:00.000 | 2023-04-04 16:29:00.000 | 1680596939949 |
| +I | 2023-04-04 16:29:00.000 | 2023-04-04 16:30:00.000 | 1680596999776 |
....

### What doesn't meet your expectations?

Full stream reading from the appendOnly table doesn't output historical full data, only the incremental data after the latest compaction was output.

### Anything else?

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the Flink 1.14 window TVF query against an append-only table using scan.mode=latest-full and the supplied snapshot history. Compare the rows read with the emitted one-minute windows; done means the query outputs complete historical window data from the earliest available snapshot, not only data after the latest compaction.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.