Race in SegmentAllocateAction with segment lock
- Dominant language
- Java
- Stars
- 14.1k
- Forks
- 3.8k
- Avg merge
- 2d 58m
- Merged PRs (30d)
- 233
Description
### Affected Version
0.16, 0.17
### Description
Before 0.16, everything was simple and there was only one type of task lock, i.e., time chunk lock. With time chunk lock, only one task can work on the same datasource and the same time period at any time. As a result, `SegmentAllocateAction` could safely assume that there is only one task asking a new segment allocation for the same datasource and the same time period.
However, with [segment lock](https://github.com/apache/druid/issues/7491), multiple tasks can work on the same datasource and the same time period simultaneously. Multiple tasks can request new segment allocations at the same time, which introduces a race condition in segment allocation.
In `SegmentAllocateAction`, it first looks for segments of overlapping intervals with the request interval populated based on the given segment granularity. This is to avoid creating segments of overlapping intervals. To do so, [it first retrieves all overlapping _used segments_ with the request interval from the metadata store](https://github.com/apache/druid/blob/master/indexing-service/src/main/java/org/apache/druid/indexing/common/actions/SegmentAllocateAction.java#L195-L196). However, this is not enough with segment lock because there could be other tasks running which created _pending segments_ of overlapping intervals. If there were such tasks, they could end up creating overlapping segments which breaks the timeline of brokers and the coordinator.
I think this check probably should include pending segments as well, and be done in `TaskLockbox` with a proper synchronization.
Contributor guide
Research direction
Start by reading indexing-service/src/main/java/org/apache/druid/indexing/common/actions/SegmentAllocateAction.java around the overlapping used-segment lookup, then inspect TaskLockbox and how pending segments are represented. Trace concurrent segment-lock allocation requests and determine how synchronization can include overlapping pending segments. Done means concurrent tasks cannot create overlapping segments that break broker and coordinator timelines.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100