Support scalable rli reading for data skipping in flink reader
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
### Task Description
**What needs to be done:**
Support more scalable RLI reading for data skipping in flink reader, i.e., flink job should schedule distributed operator in the pipeline to collect RLI records for large volumes of hoodie keys derived from the query predicate.
**Why this task is needed:**
Currently, the collecting of RLI records happen in a single process with multiple threads (either in JobManage or job client), which is suitable for cases where the amount of hoodie keys derived from query predicate is not large, but when there are multiple key fields, or the query predicate look up records with a large amount of possible keys, it may cause the collecting of RLI to run slowly. We should schedule a dedicate operator with parallel sub-tasks to collect RLI to make the data skipping more scalable and efficient.
### Task Type
Performance optimization
### Related Issues
**Parent feature issue:** (if applicable )
**Related issues:**
NOTE: Use `Relationships` button to add parent/blocking issues after issue is created.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.