Race when killing segments
- Dominant language
- Java
- Stars
- 14.1k
- Forks
- 3.8k
- Avg merge
- 2d 31m
- Merged PRs (30d)
- 209
Description
In Druid, Killing segments means removing the segment files from the deep storage and deletes the entries from the metadata store. This is currently done using KillTask. Since it's a task, it first gets taskLocks and do the below:
```java
final List unusedSegments = toolbox
.getTaskActionClient()
.submit(new SegmentListUnusedAction(getDataSource(), getInterval()));
if (!TaskActionPreconditions.isLockCoversSegments(taskLockMap, unusedSegments)) {
throw new ISE(
"Locks[%s] for task[%s] can't cover segments[%s]",
taskLockMap.values().stream().flatMap(List::stream).collect(Collectors.toList()),
getId(),
unusedSegments
);
}
// Kill segments
for (DataSegment segment : unusedSegments) {
toolbox.getDataSegmentKiller().kill(segment);
toolbox.getTaskActionClient().submit(new SegmentNukeAction(ImmutableSet.of(segment)));
}
```
However, the coordinator provides an HTTP endpoint to enable a segment (setting `used` = true for a segment) which can happen between `SegmentListUnusedAction` and `SegmentNukeAction` without getting a taskLock.
Also, the order of killing segments should be changed: the metadata store must be updated first before removing the segment file.
I think we may change the way of killing segments to not use the task. Instead, the coordinator can kill segments directly because enabling/disabling existing segments happens only in the coordinator.
Contributor guide
Research direction
Start by tracing KillTask, SegmentListUnusedAction, and SegmentNukeAction, then inspect the coordinator HTTP endpoint that re-enables segments. Define how killing prevents a segment from being re-enabled between listing and deletion, and ensure the metadata entry is removed before the deep-storage file; the issue does not name specific files or tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100