apache / apache/pinot

Only download required files from segment for metadata push

Open
#7,791 14 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
1d 21h
Merged PRs (30d)
189

Description

Currently in `SegmentPushUtils.generateSegmentMetadataFile(PinotFS fileSystem, URI tarFileURI)`, it calls
``` java
fileSystem.copyToLocalFile(tarFileURI, tarFile);
```
to download the segment from deep store and create a temp copy locally. It then unpacks and untars the segment, extracts the two files of interest (`creation.meta` and `metadata.properties`), creates a new tarball with these two files, and then pushes that to the controller.

The actual amount of data extracted from each segment is a tiny fraction of the segment's total size. e.g. For a 200MB segment, the two files of interest are about 4K bytes. When a large number of segments are being pushed, this results in a significant performance hit.

Instead, it's possible to stream/unpack the segment and extract only the data from the two target files, via something like:
``` java
String uuid = UUID.randomUUID().toString();
File segmentMetadataTarFile = new File(FileUtils.getTempDirectory(), "segmentMetadata-" + uuid + ".tar.gz");
if (segmentMetadataTarFile.exists()) {
FileUtils.forceDelete(segmentMetadataTarFile);
}
GzipCompressorOutputStream gzOut = new GzipCompressorOutputStream(new FileOutputStream(segmentMetadataTarFile));
TarArchiveOutputStream tos = new TarArchiveOutputStream(gzOut);

TarArchiveInputStream tis = new TarArchiveInputStream(new GZIPInputStream(fileSystem.open(segmentUri)));

TarArchiveEntry tarEntry;
while ((tarEntry = tis.getNextTarEntry()) != null) {
System.out.format("%s: %d\n", tarEntry.getName(), tarEntry.getRealSize());

String fullName = tarEntry.getName();
String filename = fullName.substring(fullName.lastIndexOf('/') + 1);
if (tarEntry.isFile() && (filename.equals("metadata.properties") || filename.contentEquals("creation.meta"))) {
TarArchiveEntry ae = new TarArchiveEntry(filename);
ae.setSize(tarEntry.getRealSize());
tos.putArchiveEntry(ae);
IOUtils.copy(tis, tos);
tos.closeArchiveEntry();
}
}

tos.finish();
tos.close();
tis.close();
```
The above is just example code, without exception handling, etc. but it is able to create the required tarball from a source segment.

The real performance win is because the two files of interest occur in the segment tarball before the big files (`columns.psf`, `star_tree_index`, etc) so it would be appropriate to enforce this ordering in the segment writer utility code.

Contributor guide

Open the contributing guide

Research direction

Start at SegmentPushUtils.generateSegmentMetadataFile(PinotFS fileSystem, URI tarFileURI) and inspect the segment writer utility that controls tar entry ordering. Trace fileSystem.open and the metadata entries creation.meta and metadata.properties, then verify the generated tarball contains those files without downloading the full segment before the large entries such as columns.psf and star_tree_index.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
database
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.