apache / apache/hudi

Abstract out one off operations within dag

Open
#14,740 0 comments 0 reactions 0 assignees View on GitHub
area:tests from-jira priority:medium
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

In existing test suite, we have a config called "execute_itr_count". When this is set to N for a particular node, out of 50 odd iterations, this node will be executed only on Nth iteration only. 

Use-case: 

we wish to execute clustering node on 10th iteration. but the entire dag needs to be executed for 25 iterations. 

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-1616
- Type: Test
- Epic: https://issues.apache.org/jira/browse/HUDI-3303
- Affects version(s):
- 0.9.0

---

## Comments

27/Mar/22 14:51;xushiyan;[~shivnarayan] is this done? or push to 0.12?;;;

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by locating the existing test-suite handling for the execute_itr_count configuration and tracing how a DAG node is selected for an iteration. Determine how the one-off operation should be abstracted while preserving execution of the full DAG for all iterations. Done means the clustering-node use case works at the requested iteration without changing the remaining iterations.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
testing
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.