apache / apache/druid

Problematic separation between IndexerMetadataStorageCoordinator and MetadataSegmentManager abstractions

Open
#8,187 0 comments 0 reactions 0 assignees View on GitHub
Area - Metadata Contributions Welcome Refactoring
Dominant language
Java
Stars
14.1k
Forks
3.8k
Avg merge
2d 58m
Merged PRs (30d)
233

Description

Several methods from `IndexerMetadataStorageCoordinator`, namely
- `getUsedSegmentsForInterval`
- `getAllUsedSegments`
- `getUsedSegmentsAndCreatedDates`
- `getUsedSegmentsForIntervals`
- `getveUnusedSegmentsForInterval`

Duplicate the `MetadataSegmentManager`'s functionality and look like much better suited for the latter class.

The conceptual difference is that `MetadataSegmentManager` caches the segments' data in memory, while `IndexerMetadataStorageCoordinator` always accesses the metadata store. This difference should be retained, but there should be comments to those methods, as well as parallel methods returning cached metadata with warnings and guidance regarding which method to use. For example, `MaterializedViewSupervisor` uses the methods from `IndexerMetadataStorageCoordinator` which seems unnecessary to me (could use the methods from `MetadataSegmentManager` instead).

If we decide that "cached metadata - direct access to metadata store" is the separation principle between `IndexerMetadataStorageCoordinator` and `MetadataSegmentManager`, then this fact should be reflected in these interfaces' names, Javadocs, and `retrieveAllDataSourceNames()` method should be moved from `MetadataSegmentManager` to `IndexerMetadataStorageCoordinator`.

Contributor guide

Open the contributing guide

Research direction

Start by comparing the IndexerMetadataStorageCoordinator and MetadataSegmentManager abstractions and their listed segment methods, then inspect MaterializedViewSupervisor's usage. Clarify whether the intended boundary is cached metadata versus direct metadata-store access. Done means the chosen separation is consistently reflected in method placement, interface names, Javadocs, and guidance for callers.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, databases
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.