[Feature] ignore expiration check interval of partition in batch mode.
- Dominant language
- Java
- Stars
- 3.4k
- Forks
- 1.4k
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 396
Description
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/incubator-paimon/issues) and found nothing similar.
### Motivation
If the execution time of a batch job is short, and it's less than the interval for partition expiration checks, the partition expiration operations will not be executed.
### Solution
During the commit phase of batch jobs, ignore expiration check interval of partition.
### Anything else?
_No response_
### Are you willing to submit a PR?
- [X] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing the batch job commit phase and the partition expiration operations, focusing on where the expiration check interval is applied. Done means batch commits perform partition expiration without being skipped because the job ran for less than that interval.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100