Make sure Spark RDDs triggering actual FS activity are only dereferenced once
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
Currently, RDD `collect()` operation is treated quite loosely and there are multiple flows which used to dereference RDDs (for ex, through `collect`, `count`, etc) that way triggering the same operations being carried out multiple times, occasionally duplicating the output already persisted on FS.
Check out HUDI-3370 for recent example.
NOTE: Even though Spark caching is supposed to make sure that we aren't writing to FS multiple times, we can't solely rely on caching to guarantee exactly once execution.
Instead, we should make sure that RDDs are only dereferenced {*}once{*}, w/in "commit" operation and all the other operations are only relying on _derivative_ data.
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-3397
- Type: Bug
- Epic: https://issues.apache.org/jira/browse/HUDI-3249
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reviewing the commit operation and the collect/count flows described in the issue, then compare them with the HUDI-3370 example. Trace where RDDs are dereferenced and identify which operations use derivative data instead. Done means FS-triggering RDDs are dereferenced only once during commit, without relying solely on Spark caching.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- data-engineering, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100