Abstract out one off operations within dag
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
In existing test suite, we have a config called "execute_itr_count". When this is set to N for a particular node, out of 50 odd iterations, this node will be executed only on Nth iteration only.
Use-case:
we wish to execute clustering node on 10th iteration. but the entire dag needs to be executed for 25 iterations.
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-1616
- Type: Test
- Epic: https://issues.apache.org/jira/browse/HUDI-3303
- Affects version(s):
- 0.9.0
---
## Comments
27/Mar/22 14:51;xushiyan;[~shivnarayan] is this done? or push to 0.12?;;;
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by locating the existing test-suite handling for the execute_itr_count configuration and tracing how a DAG node is selected for an iteration. Determine how the one-off operation should be abstracted while preserving execution of the full DAG for all iterations. Done means the clustering-node use case works at the requested iteration without changing the remaining iterations.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- testing
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100