apache / apache/paimon

[Feature] paimon_incremental_to_auto_tag return whole data if earlier tag does not exist

Open
#5,094 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.

### Motivation

The tables with pk are ingested into paimon with batch processing tasks initialy, then ingested in real time. And the real time operation can create tags periodically.
For the real time part, we could use paimon incremental query to get the difference between tags, and process them incrementally.
The initial data, on the other hand, could not be retrieved through the incremental query. We have to use a total different way, like selecting all data without filter condition. That increase the complexity of data processing method.

### Solution

The would be better if the paimon incremental query returns the whole data when missing earlier tags. If so, we could use the same method to process the initial part and real time part.
```sql
-- If the tags earlier than 2024-12-04 doesn't exist, return whole data instead of empty.
SELECT * FROM paimon_incremental_to_auto_tag('tableName', '2024-12-04');
```

### Anything else?

_No response_

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

No files or tests are named. Start by locating the paimon_incremental_to_auto_tag implementation and reading how it handles a requested tag with no earlier tag; done means the example query returns the whole table instead of an empty result when the earlier tag is missing, while existing incremental behavior remains intact.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, sql
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.