apache / apache/druid

Race when killing segments

Open
#6,816 3 comments 0 reactions 0 assignees View on GitHub
Bug stale
Dominant language
Java
Stars
14.1k
Forks
3.8k
Avg merge
2d 31m
Merged PRs (30d)
209

Description

In Druid, Killing segments means removing the segment files from the deep storage and deletes the entries from the metadata store. This is currently done using KillTask. Since it's a task, it first gets taskLocks and do the below:

```java
final List unusedSegments = toolbox
.getTaskActionClient()
.submit(new SegmentListUnusedAction(getDataSource(), getInterval()));

if (!TaskActionPreconditions.isLockCoversSegments(taskLockMap, unusedSegments)) {
throw new ISE(
"Locks[%s] for task[%s] can't cover segments[%s]",
taskLockMap.values().stream().flatMap(List::stream).collect(Collectors.toList()),
getId(),
unusedSegments
);
}

// Kill segments
for (DataSegment segment : unusedSegments) {
toolbox.getDataSegmentKiller().kill(segment);
toolbox.getTaskActionClient().submit(new SegmentNukeAction(ImmutableSet.of(segment)));
}
```

However, the coordinator provides an HTTP endpoint to enable a segment (setting `used` = true for a segment) which can happen between `SegmentListUnusedAction` and `SegmentNukeAction` without getting a taskLock.

Also, the order of killing segments should be changed: the metadata store must be updated first before removing the segment file.

I think we may change the way of killing segments to not use the task. Instead, the coordinator can kill segments directly because enabling/disabling existing segments happens only in the coordinator.

Contributor guide

Open the contributing guide

Research direction

Start by tracing KillTask, SegmentListUnusedAction, and SegmentNukeAction, then inspect the coordinator HTTP endpoint that re-enables segments. Define how killing prevents a segment from being re-enabled between listing and deletion, and ensure the metadata entry is removed before the deep-storage file; the issue does not name specific files or tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.