apache / apache/druid

MetadataSegmentView should correctly handle the situation of Coordinator being down or changed leadership

Open
#7,391 3 comments 0 reactions 0 assignees View on GitHub
Area - Metadata Area - SQL
Dominant language
Java
Stars
14.1k
Forks
3.8k
Avg merge
2d 58m
Merged PRs (30d)
233

Description

Regardless of whether any of the designs proposed for `MetadataSegmentView` here: https://github.com/apache/incubator-druid/issues/6834#issuecomment-478569365 is applied, `MetadataSegmentView` should document its behavior in the face of the Coordinator being down, or just started, or leadership changed.

Currently, there is a single comment ["Note that coordinator must be up to get segments"](https://github.com/apache/incubator-druid/pull/6901/files#diff-5707cd19ee7a43a258f29e61fd385d4cR181) which, IMO, only adds FUD and doesn't explain anything.

Ideally, `MetadataSegmentView` should correctly recognize the situations of changed Coordinator's leadership and not to flush all cached published segments.

Explanation may look like the following:
```
/**
* When Coordinator is just started or leadership changed,
* its SegmentsMetadata is unpopulated. Then the endpoint /metadata/abc/foo/bar
* which we are querying returns code XXX (see {@link DataSourcesResource#theRelevantMethod}).
* We process this code here and don't flush {@link #cachedPublishedSegments} in this case.
*/
```

FYI @surekhasaharan

Contributor guide

Open the contributing guide

Research direction

Start by locating MetadataSegmentView and DataSourcesResource#theRelevantMethod, then read the linked discussion in issue 6834 and the referenced change in pull request 6901. Trace the response used when the Coordinator is starting or has changed leadership, and verify that the documented and implemented behavior preserves cachedPublishedSegments rather than flushing them unnecessarily.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.