Configuring ingestion to default to pushing metadata so controllers do not always pull segments for metadata
Open
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 189
Description
From Slack:
> joey: the controller is pulling the entire segment. Does that happen for every segment? Is it just pulling it to grab the metadata, or does it persist additional data?
> That seems like a gnarly bottleneck if we're pushing TBs of data from a Spark app.
> kishore: its just pulling it for metadata
> there is a way to push the metadata as well
> so that it does not pull the entire segment
> we should make that the default
cc @kishoreg
Contributor guide
Assessment
This issue has not been assessed yet.