Fix graceful shutdown for tasks
- Dominant language
- Java
- Stars
- 14.1k
- Forks
- 3.8k
- Avg merge
- 2d 58m
- Merged PRs (30d)
- 233
Description
### Affected Version
0.14, 0.15, 0.16
### Description
Since https://github.com/apache/incubator-druid/pull/6828, [task shutdown is always the graceful shutdown](https://github.com/apache/incubator-druid/blob/master/indexing-service/src/main/java/org/apache/druid/indexing/overlord/SingleTaskBackgroundRunner.java#L184-L186). This was for some task types to clean up their resources. For example, Hadoop task kills its Hadoop job when it's stopped (https://github.com/apache/incubator-druid/pull/6828). The parallel indexing task kills its all running sub tasks (https://github.com/apache/incubator-druid/pull/7041).
However, this is different from how the stream ingestion task had been using graceful shutdown. On graceful shutdown, they immediately start persisting all segments in memory onto disk and publishing segments before they stop. This is to create a checkpoint when they stop, not for resource cleanup. As a result, https://github.com/apache/incubator-druid/pull/6828 unexpectedly changed the behavior of Kafka/Kinesis indexing service when the supervisor kills tasks.
I think we need to distinguish immediate stop and graceful stop again. Immediate stop always involves resource cleanup. Graceful stop may involve extra work in addition to necessary resource cleanup.
Contributor guide
Research direction
Start with SingleTaskBackgroundRunner.java and compare the shutdown behavior introduced by PR 6828 with the task behavior described in PRs 7041 and 6828. Trace the stream ingestion task's supervisor stop path for Kafka/Kinesis tasks. Done means immediate stop performs resource cleanup while graceful stop also preserves the segment checkpointing behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, kafka
- Domain
- distributed-systems, stream-processing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100