apache / apache/druid

Race in SegmentAllocateAction with segment lock

Open
#9,571 0 comments 1 reaction 0 assignees View on GitHub
Bug
Dominant language
Java
Stars
14.1k
Forks
3.8k
Avg merge
2d 58m
Merged PRs (30d)
233

Description

### Affected Version

0.16, 0.17

### Description

Before 0.16, everything was simple and there was only one type of task lock, i.e., time chunk lock. With time chunk lock, only one task can work on the same datasource and the same time period at any time. As a result, `SegmentAllocateAction` could safely assume that there is only one task asking a new segment allocation for the same datasource and the same time period.

However, with [segment lock](https://github.com/apache/druid/issues/7491), multiple tasks can work on the same datasource and the same time period simultaneously. Multiple tasks can request new segment allocations at the same time, which introduces a race condition in segment allocation.

In `SegmentAllocateAction`, it first looks for segments of overlapping intervals with the request interval populated based on the given segment granularity. This is to avoid creating segments of overlapping intervals. To do so, [it first retrieves all overlapping _used segments_ with the request interval from the metadata store](https://github.com/apache/druid/blob/master/indexing-service/src/main/java/org/apache/druid/indexing/common/actions/SegmentAllocateAction.java#L195-L196). However, this is not enough with segment lock because there could be other tasks running which created _pending segments_ of overlapping intervals. If there were such tasks, they could end up creating overlapping segments which breaks the timeline of brokers and the coordinator.

I think this check probably should include pending segments as well, and be done in `TaskLockbox` with a proper synchronization.

Contributor guide

Open the contributing guide

Research direction

Start by reading indexing-service/src/main/java/org/apache/druid/indexing/common/actions/SegmentAllocateAction.java around the overlapping used-segment lookup, then inspect TaskLockbox and how pending segments are represented. Trace concurrent segment-lock allocation requests and determine how synchronization can include overlapping pending segments. Done means concurrent tasks cannot create overlapping segments that break broker and coordinator timelines.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.