apache / apache/pinot

Support in-memory segment generation and upload

Open
#7,434 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
2d 3h
Merged PRs (30d)
195

Description

Currently the SegmentWriter and SegmentUploader writes data into a TAR file and then upload to Pinot servers.

Can we optimize the write to disk by creating a in-memory TAR file format to speed up the upload?

Contributor guide

Open the contributing guide

Research direction

Start by reading SegmentWriter and SegmentUploader to trace how data is written into a TAR file and uploaded to Pinot servers. Define the in-memory TAR approach and compare it with the current disk-based flow; done means segment generation and upload work without the intermediate disk write while preserving existing behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.