Support cow flink batch create fs input split asynchronously
Open
area:incr-processing
area:performance
area:sql
engine:flink
from-jira
priority:blocker
status:pr-available
type:improvement
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 112
Description
*CopyOnWriteInputFormat#createInputSplits* is invoked by *org.apache.flink.runtime.executiongraph.ExecutionJobVertex* in JobManager to create file input splits synchronously.
It's found that in batch mode this will occupy the largest share of job submission time.
So in this PR it will be optimized by creating input splits in thread pool executor asynchronously.
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-5068
- Type: Improvement
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.