apache / apache/druid

Optimize coordinator API to retrieve segments with overshadowed status

Open
#7,571 24 comments 0 reactions 0 assignees View on GitHub
Area - Metadata Performance
Dominant language
Java
Stars
14.1k
Forks
3.8k
Avg merge
2d 58m
Merged PRs (30d)
233

Description

Coordinator API `/druid/coordinator/v1/metadata/segments?includeOvershadowedStatus` builds `VersionedIntervalTimeline` which can be memory intensive and expensive if this API is called multiple times but multiple brokers in a cluster.

The comment [here](https://github.com/apache/incubator-druid/pull/7425#discussion_r279458071) suggests :

- `isOvershadowed` becomes a non-final field of `DataSegment` object itself, not participating in `equals()` and `hashCode()`.
- Add `interface SegmentsAccess { ImmutableDruidDataSource prepare(String dataSource); Iterable iterateAll(); }` (strawman naming)
- Add `DataSourceAccess computeOvershadowed()` method to `SQLSegmentMetadataManager`, which performs this computation for every snapshot of `SQLSegmentMetadataManager.dataSources` (which is updated in `poll()`) *at most once, lazily*.
- Both endpoints in `MetadataResource` and Coordination balancing logic (which currently computes isOvershadowed status on its own, too) use this API.
- On the side of `MetadataSegmentView`, maintain something like a `Map` and update overshadowed status like `map.get(segmentFromCoordinator).setOvershadowed(segmentFromCoordinator.isOvershadowed())`

Contributor guide

Open the contributing guide

Research direction

Start by reading MetadataResource, SQLSegmentMetadataManager, MetadataSegmentView, DataSegment, and the coordination balancing logic that currently computes overshadowed status. Trace how VersionedIntervalTimeline is built for the metadata endpoint and how dataSources are updated in poll(). Done means both metadata endpoints and coordination logic share lazy, at-most-once snapshot computation without repeated timeline construction.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
api, backend, databases
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.