apache / apache/pinot

Inconsistent segment file extension for TarGzPush and MetadataPush

Open
#10,375 6 comments 0 reactions 1 assignee Assigned to @deepthi912 View on GitHub
good first issue
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
1d 21h
Merged PRs (30d)
189

Description

Our SegmentGenerationJobRunner adds `tar.gz` extension to the segment file name.

- TarGzPush: the controller side segment upload logic removes `tar.gz`extension when copying data to deep storage. The segment file will be stored as `fileName`.
- MetadataPush: we use pinotFS to copy file from local to dst. Segment file will be stored as `fileName.tar.gz`.

It's going to be good if we can make the name convension the same. But, this will change the file name convension to the deepstorage. We should at least consider to add the config to remove the extension to have the clear migration path. (e.g. some people can start the cluster using TarGzPush and want to migrate to MetadataPush and keep the file names the same convention in the deep storage)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.