Prevent dag trigger in mappartitions if possible
Open
engine:spark
from-jira
priority:high
type:improvement
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
mapparitions gives and iterator and expects to return an iterator. If iterator.isEmpty is called, that can cause the dag to start processing
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-7525
- Type: Improvement
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by locating the mapPartitions implementation and tracing how its iterator is consumed, especially any iterator.isEmpty calls. Check the relevant execution path and tests, if present, to understand when the DAG begins processing. Done means avoiding an unnecessary DAG trigger from the mapPartitions iterator without breaking its expected iterator behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 45/100