apache / apache/paimon

[Feature] ignore expiration check interval of partition in batch mode.

Open
#2,969 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/incubator-paimon/issues) and found nothing similar.

### Motivation

If the execution time of a batch job is short, and it's less than the interval for partition expiration checks, the partition expiration operations will not be executed.

### Solution

During the commit phase of batch jobs, ignore expiration check interval of partition.

### Anything else?

_No response_

### Are you willing to submit a PR?

- [X] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing the batch job commit phase and the partition expiration operations, focusing on where the expiration check interval is applied. Done means batch commits perform partition expiration without being skipped because the job ran for less than that interval.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.