apache / apache/druid

Fix graceful shutdown for tasks

Open
#8,718 4 comments 0 reactions 0 assignees View on GitHub
Area - Batch Ingestion Area - Streaming Ingestion Bug
Dominant language
Java
Stars
14.1k
Forks
3.8k
Avg merge
2d 58m
Merged PRs (30d)
233

Description

### Affected Version

0.14, 0.15, 0.16

### Description

Since https://github.com/apache/incubator-druid/pull/6828, [task shutdown is always the graceful shutdown](https://github.com/apache/incubator-druid/blob/master/indexing-service/src/main/java/org/apache/druid/indexing/overlord/SingleTaskBackgroundRunner.java#L184-L186). This was for some task types to clean up their resources. For example, Hadoop task kills its Hadoop job when it's stopped (https://github.com/apache/incubator-druid/pull/6828). The parallel indexing task kills its all running sub tasks (https://github.com/apache/incubator-druid/pull/7041).

However, this is different from how the stream ingestion task had been using graceful shutdown. On graceful shutdown, they immediately start persisting all segments in memory onto disk and publishing segments before they stop. This is to create a checkpoint when they stop, not for resource cleanup. As a result, https://github.com/apache/incubator-druid/pull/6828 unexpectedly changed the behavior of Kafka/Kinesis indexing service when the supervisor kills tasks.

I think we need to distinguish immediate stop and graceful stop again. Immediate stop always involves resource cleanup. Graceful stop may involve extra work in addition to necessary resource cleanup.

Contributor guide

Open the contributing guide

Research direction

Start with SingleTaskBackgroundRunner.java and compare the shutdown behavior introduced by PR 6828 with the task behavior described in PRs 7041 and 6828. Trace the stream ingestion task's supervisor stop path for Kafka/Kinesis tasks. Done means immediate stop performs resource cleanup while graceful stop also preserves the segment checkpointing behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, kafka
Domain
distributed-systems, stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.