apache / apache/paimon

[Bug] Batch increment query has empty result when overwrite

Open
#2,803 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/incubator-paimon/issues) and found nothing similar.

### Paimon version

0.7

### Compute Engine

flink1.17

### Minimal reproduce step

when i batch insert overwrite paimon table , i can not use Batch increment query for any result.

```
CREATE TABLE if not EXISTS use_be_hours_2 (
user_id BIGINT,
item_id BIGINT,
behavior STRING,
dt STRING,
hh STRING,
PRIMARY KEY (user_id) NOT ENFORCED
)
with (
'manifest.format'='orc',
'changelog-producer' = 'input'
);

INSERT INTO use_be_hours_2
VALUES
(1, 1, 'watch', '2022-01-01', '10'),
(2, 2, 'like', '2022-01-01', '10');

INSERT OVERWRITE use_be_hours_2
VALUES
(1, 100, 'watch', '2022-01-01', '10'),
(3, 100, 'like', '2022-01-01', '10');

INSERT OVERWRITE use_be_hours_2
VALUES
(1, 200, 'watch', '2022-01-01', '10'),
(3, 200, 'like', '2022-01-01', '10'),
(4, 200, 'like', '2022-01-01', '10')
;
```
my query:
```
SELECT * FROM use_be_hours_2 /*+ OPTIONS('incremental-between' = '1,3') */;

SELECT * FROM use_be_hours_2$audit_log /*+ OPTIONS('incremental-between' = '1,3') */;
```

the result above both are empty , is it normal ?

### What doesn't meet your expectations?

the result above both are empty , is it normal ?

### Anything else?

_No response_

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by running the provided Flink 1.17 SQL reproduction against Paimon 0.7, including both the table query and the audit_log query with incremental-between set to 1,3. Compare the results after the batch insert and two INSERT OVERWRITE operations; done means determining whether empty results are expected and, if not, identifying the affected incremental-query behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
sql
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.