apache / apache/druid

Druid did not remove segments from middle manager after ingestion task failed

Open
#13,183 2 comments 0 reactions 0 assignees View on GitHub
Area - Ingestion Bug
Dominant language
Java
Stars
14.1k
Forks
3.8k
Avg merge
2d 58m
Merged PRs (30d)
233

Description

### Affected Version
0.22.1

### Description
**What happened?**
- Druid did not remove segments from middle manager after the ingestion task failed

Background
- For some reason, some Kafka ingestion tasks at some point failed. We saw InterruptedExceptions in the logs.
- We are thinking it is likely due to some overlord failure.
- The tasks reran and they auto recovered.

Question
- One of the ingestion task that failed may have created some realtime segments that got interrupted. Those segments did not get removed from middle manager.
- Is this behavior expected? If the tasks auto recovered, such stray segments should also have been deleted but they continued to stay on middle manager.
![RealtimeSegments](https://user-images.githubusercontent.com/15020965/193903020-84c0e7e4-0040-47ba-8ea1-4a17c1ff6252.png)

Contributor guide

Open the contributing guide

Research direction

Begin with the middle manager's handling of failed Kafka ingestion tasks and the Overlord/task logs, especially the InterruptedException path. Confirm whether interrupted realtime segments are expected to be removed after recovery and document or reproduce the cleanup outcome; no file or test is identified in the issue.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, kafka
Domain
backend, data, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.