[SUPPORT] split_reader don't checkpoint before consuming all splits
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
**split_reader don't checkpoint before consuming all splits**
- Have you gone through our [FAQs](https://hudi.apache.org/learn/faq/)?
- Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org.
- If you have triaged this as a bug, then file an [issue](https://issues.apache.org/jira/projects/HUDI/issues) directly.
**Describe the problem you faced**
When using flink to incrementally query a mor table with many splits(read.start-commit=earliest),The first checkpoint of the reader is successful after all splits are consumed. It takes a lot of time to cause checkpoint timeout. In my understanding, the process of reading data based on mini batch can trigger and complete the checkpoint.

**To Reproduce**
Steps to reproduce the behavior:
1. mor table with many splits
2. read.start-commit=ealiest
3. read.tasks=1
4. checkpoint interval=10s and checkpoint timeout=10min
**Environment Description**
* Hudi version : 0.11.1
* flink version : 1.14.5
**Stacktrace**
`Caused by: org.apache.flink.util.FlinkRuntimeException: Exceeded checkpoint tolerable failure threshold.`
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.