apache / apache/druid

VersionedIntervalTimeline performance corner case with high number of overlapping segments

Open
#8,846 4 comments 0 reactions 0 assignees View on GitHub
Area - Segment Balancing/Coordination Discuss Performance
Dominant language
Java
Stars
14.1k
Forks
3.8k
Avg merge
2d 58m
Merged PRs (30d)
233

Description

Recently all of our historical nodes restarted on a cluster which was serving about 50000 segments of following nature.
each segment's interval would be 24 hours and each successive segment overlaps with previous one for 1439 minutes (1440 minutes is 24 hours), for example segment intervals and versions might look like

2019-01-01T00:00:00.000Z - 2019-01-01T23:59:00.000Z , v1
2019-01-01T00:01:00.000Z - 2019-01-02T00:00:00.000Z , v2
2019-01-01T00:02:00.000Z - 2019-01-02T00:01:00.000Z , v3
2019-01-01T00:03:00.000Z - 2019-01-02T00:02:00.000Z , v4
...
...

that triggered a sequence of `VersionIntervalTimeline.remove(..)` calls for each segment one by one and broker/coordinator never recovered and needed a forced restart because `VersionIntervalTimeline.remove(..)` becomes very expensive for above scenario and never finished.

I did a quick prototype to batch multiple `VersionIntervalTimeline.remove(..)` calls into a single `VersionIntervalTimeline.removeAll(..)` call which could be used when data servers go down which had few optimizations possible. Batched call would first remove all entries from `allTimelineEntries` and then from `complete/incompletePartitionTimeline` and then adjust them based on the state of `allTimelineEntries` , with batched version `allTimelineEntries` has significantly fewer entries and no unnecessary corrections are to be made to `complete/incompletePartitionTimeline` which happens in non-batched removals.

..creating this issue to discuss other proposed solutions.

Contributor guide

Open the contributing guide

Research direction

Start with VersionIntervalTimeline.remove(..) and the proposed removeAll(..) prototype described in the issue, tracing how removals update allTimelineEntries and the complete/incompletePartitionTimeline. Compare behavior using the 50,000 heavily overlapping segments described; done means the removal sequence completes without the broker or coordinator requiring a forced restart.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases, performance
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.