apache / apache/hudi

Make sure Spark RDDs triggering actual FS activity are only dereferenced once

Open
#15,017 1 comment 0 reactions 0 assignees View on GitHub
area:writer engine:spark from-jira priority:blocker priority:critical status:pr-available type:bug
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

Currently, RDD `collect()` operation is treated quite loosely and there are multiple flows which used to dereference RDDs (for ex, through `collect`, `count`, etc) that way triggering the same operations being carried out multiple times, occasionally duplicating the output already persisted on FS.

Check out HUDI-3370 for recent example.

NOTE: Even though Spark caching is supposed to make sure that we aren't writing to FS multiple times, we can't solely rely on caching to guarantee exactly once execution.

Instead, we should make sure that RDDs are only dereferenced {*}once{*}, w/in "commit" operation and all the other operations are only relying on _derivative_ data.

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-3397
- Type: Bug
- Epic: https://issues.apache.org/jira/browse/HUDI-3249

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reviewing the commit operation and the collect/count flows described in the issue, then compare them with the HUDI-3370 example. Trace where RDDs are dereferenced and identify which operations use derivative data instead. Done means FS-triggering RDDs are dereferenced only once during commit, without relying solely on Spark caching.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
data-engineering, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.