apache / apache/pinot

Configuring ingestion to default to pushing metadata so controllers do not always pull segments for metadata

Open
#5,879 2 comments 0 reactions 1 assignee Claimed by @xiangfu0 View on GitHub
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
1d 21h
Merged PRs (30d)
189

Description

From Slack:
> joey: the controller is pulling the entire segment. Does that happen for every segment? Is it just pulling it to grab the metadata, or does it persist additional data?
> That seems like a gnarly bottleneck if we're pushing TBs of data from a Spark app.

> kishore: its just pulling it for metadata
> there is a way to push the metadata as well
> so that it does not pull the entire segment
> we should make that the default

cc @kishoreg

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.