apache / apache/druid

[Proposal] Kill Hadoop MR task on kill of ingestion task and resume ability for Hadoop ingestion tasks

Open
#6,803 6 comments 2 reactions 0 assignees View on GitHub
Area - Batch Ingestion Proposal stale
Dominant language
Java
Stars
14.1k
Forks
3.8k
Avg merge
2d 31m
Merged PRs (30d)
209

Description

We plan to implement these features in a two phase solution:

**Phase I** : _Implement the kill task feature_

Currently on killing the Hadoop ingestion task from the overlord ui does not kill the MR job resulting in unnecessary wastage of resources.

The way we are thinking of doing this is by writing the job id of the MR job in a file `mapReduceJobId.json` which will be stored under the `taskBaseDir`
A separate file will be required as the MR job is executed in different JVM and there is no way it could communicate with kill task snippet, so writing the required info in a file was the only option
Currently, this file will only store the running JobID, this way when some one wishes to kill the ingestion task it can read the current running MR job (if any) and issue a yarn kill command

**Phase II**: _Implement the resume ability for Hadoop ingestion tasks_

The above file will now store the job Id of each intermediate MR task so that we can track till what step the ingestion was completed and can resume from the following step instead of executing from the beginning

Contributor guide

Open the contributing guide

Research direction

Start by tracing Hadoop ingestion tasks from the Overlord UI and identify where the running MapReduce job ID can be written under taskBaseDir. Review how the separate JVM executes the job and how a kill request can issue a YARN kill command; done means Phase I stops the active job and Phase II records intermediate job IDs so ingestion can resume from the next step.

Written by the indexing model from the issue text.

Assessment

Tech stack
hadoop, java
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.