VersionedIntervalTimeline performance corner case with high number of overlapping segments
- Dominant language
- Java
- Stars
- 14.1k
- Forks
- 3.8k
- Avg merge
- 2d 58m
- Merged PRs (30d)
- 233
Description
Recently all of our historical nodes restarted on a cluster which was serving about 50000 segments of following nature.
each segment's interval would be 24 hours and each successive segment overlaps with previous one for 1439 minutes (1440 minutes is 24 hours), for example segment intervals and versions might look like
2019-01-01T00:00:00.000Z - 2019-01-01T23:59:00.000Z , v1
2019-01-01T00:01:00.000Z - 2019-01-02T00:00:00.000Z , v2
2019-01-01T00:02:00.000Z - 2019-01-02T00:01:00.000Z , v3
2019-01-01T00:03:00.000Z - 2019-01-02T00:02:00.000Z , v4
...
...
that triggered a sequence of `VersionIntervalTimeline.remove(..)` calls for each segment one by one and broker/coordinator never recovered and needed a forced restart because `VersionIntervalTimeline.remove(..)` becomes very expensive for above scenario and never finished.
I did a quick prototype to batch multiple `VersionIntervalTimeline.remove(..)` calls into a single `VersionIntervalTimeline.removeAll(..)` call which could be used when data servers go down which had few optimizations possible. Batched call would first remove all entries from `allTimelineEntries` and then from `complete/incompletePartitionTimeline` and then adjust them based on the state of `allTimelineEntries` , with batched version `allTimelineEntries` has significantly fewer entries and no unnecessary corrections are to be made to `complete/incompletePartitionTimeline` which happens in non-batched removals.
..creating this issue to discuss other proposed solutions.
Contributor guide
Research direction
Start with VersionIntervalTimeline.remove(..) and the proposed removeAll(..) prototype described in the issue, tracing how removals update allTimelineEntries and the complete/incompletePartitionTimeline. Compare behavior using the 50,000 heavily overlapping segments described; done means the removal sequence completes without the broker or coordinator requiring a forced restart.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases, performance
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100