apache / apache/hudi

Make sure Hudi Spark relations implementations provide similar file-scanning metrics

Open
#15,114 1 comment 0 reactions 0 assignees View on GitHub
from-jira priority:critical type:improvement
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

After rebasing Hudi's COW table onto `BaseFileOnlyRelation` from `HadoopFsRelation`, we lost very helpful file-scanning metrics like # of files scanned, total size of files scanned, etc.

 

After some investigation, i've found out that this occurred b/c now instead of relying on `FileScan` node in Spark plans like we're before, it now uses `DataScan` node that doesn't provide such metrics.

This unwanted transition occurred b/c Spark internally predicates on `HadoopFsRelation` to decide whether it's a `FileScan` or `DataScan`, and since we stopped using `HadoopFsRelation` Hudi relations now fall into the latter bucket

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-3882
- Type: Improvement
- Epic: https://issues.apache.org/jira/browse/HUDI-1297

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing Hudi's COW table relation implementation and how Spark chooses between FileScan and DataScan for HadoopFsRelation. Compare the available file-scanning metrics and determine how Hudi relations can provide equivalent metrics; done means plans expose metrics such as files scanned and total scanned-file size again.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
data-engineering, distributed-systems, performance
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.