MoR Incremental count queries trigger full scan of files in table
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
While running the `TestMORDataSource` datasource tests I saw that we eventually call `HoodiePruneFileSourcePartitions` which will list all of the files in the table instead of the files that are relevant to the incremental query. Ideally this would be limited to the files that were impacted by commits within the range specified.
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-6807
- Type: Bug
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by running the TestMORDataSource datasource tests and trace the call to HoodiePruneFileSourcePartitions. Inspect how the incremental query’s commit range is passed into file pruning. Done means incremental count queries inspect only files impacted by commits in the specified range rather than listing the entire table.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- data-engineering, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100