Make sure Hudi Spark relations implementations provide similar file-scanning metrics
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
After rebasing Hudi's COW table onto `BaseFileOnlyRelation` from `HadoopFsRelation`, we lost very helpful file-scanning metrics like # of files scanned, total size of files scanned, etc.
After some investigation, i've found out that this occurred b/c now instead of relying on `FileScan` node in Spark plans like we're before, it now uses `DataScan` node that doesn't provide such metrics.
This unwanted transition occurred b/c Spark internally predicates on `HadoopFsRelation` to decide whether it's a `FileScan` or `DataScan`, and since we stopped using `HadoopFsRelation` Hudi relations now fall into the latter bucket
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-3882
- Type: Improvement
- Epic: https://issues.apache.org/jira/browse/HUDI-1297
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing Hudi's COW table relation implementation and how Spark chooses between FileScan and DataScan for HadoopFsRelation. Compare the available file-scanning metrics and determine how Hudi relations can provide equivalent metrics; done means plans expose metrics such as files scanned and total scanned-file size again.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- data-engineering, distributed-systems, performance
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100