Optimize coordinator API to retrieve segments with overshadowed status
- Dominant language
- Java
- Stars
- 14.1k
- Forks
- 3.8k
- Avg merge
- 2d 58m
- Merged PRs (30d)
- 233
Description
Coordinator API `/druid/coordinator/v1/metadata/segments?includeOvershadowedStatus` builds `VersionedIntervalTimeline` which can be memory intensive and expensive if this API is called multiple times but multiple brokers in a cluster.
The comment [here](https://github.com/apache/incubator-druid/pull/7425#discussion_r279458071) suggests :
- `isOvershadowed` becomes a non-final field of `DataSegment` object itself, not participating in `equals()` and `hashCode()`.
- Add `interface SegmentsAccess { ImmutableDruidDataSource prepare(String dataSource); Iterable iterateAll(); }` (strawman naming)
- Add `DataSourceAccess computeOvershadowed()` method to `SQLSegmentMetadataManager`, which performs this computation for every snapshot of `SQLSegmentMetadataManager.dataSources` (which is updated in `poll()`) *at most once, lazily*.
- Both endpoints in `MetadataResource` and Coordination balancing logic (which currently computes isOvershadowed status on its own, too) use this API.
- On the side of `MetadataSegmentView`, maintain something like a `Map` and update overshadowed status like `map.get(segmentFromCoordinator).setOvershadowed(segmentFromCoordinator.isOvershadowed())`
Contributor guide
Research direction
Start by reading MetadataResource, SQLSegmentMetadataManager, MetadataSegmentView, DataSegment, and the coordination balancing logic that currently computes overshadowed status. Trace how VersionedIntervalTimeline is built for the metadata endpoint and how dataSources are updated in poll(). Done means both metadata endpoints and coordination logic share lazy, at-most-once snapshot computation without repeated timeline construction.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- api, backend, databases
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100