[Proposal] Kill Hadoop MR task on kill of ingestion task and resume ability for Hadoop ingestion tasks
- Dominant language
- Java
- Stars
- 14.1k
- Forks
- 3.8k
- Avg merge
- 2d 31m
- Merged PRs (30d)
- 209
Description
We plan to implement these features in a two phase solution:
**Phase I** : _Implement the kill task feature_
Currently on killing the Hadoop ingestion task from the overlord ui does not kill the MR job resulting in unnecessary wastage of resources.
The way we are thinking of doing this is by writing the job id of the MR job in a file `mapReduceJobId.json` which will be stored under the `taskBaseDir`
A separate file will be required as the MR job is executed in different JVM and there is no way it could communicate with kill task snippet, so writing the required info in a file was the only option
Currently, this file will only store the running JobID, this way when some one wishes to kill the ingestion task it can read the current running MR job (if any) and issue a yarn kill command
**Phase II**: _Implement the resume ability for Hadoop ingestion tasks_
The above file will now store the job Id of each intermediate MR task so that we can track till what step the ingestion was completed and can resume from the following step instead of executing from the beginning
Contributor guide
Research direction
Start by tracing Hadoop ingestion tasks from the Overlord UI and identify where the running MapReduce job ID can be written under taskBaseDir. Review how the separate JVM executes the job and how a kill request can issue a YARN kill command; done means Phase I stops the active job and Phase II records intermediate job IDs so ingestion can resume from the next step.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- hadoop, java
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100