Only download required files from segment for metadata push
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 189
Description
Currently in `SegmentPushUtils.generateSegmentMetadataFile(PinotFS fileSystem, URI tarFileURI)`, it calls
``` java
fileSystem.copyToLocalFile(tarFileURI, tarFile);
```
to download the segment from deep store and create a temp copy locally. It then unpacks and untars the segment, extracts the two files of interest (`creation.meta` and `metadata.properties`), creates a new tarball with these two files, and then pushes that to the controller.
The actual amount of data extracted from each segment is a tiny fraction of the segment's total size. e.g. For a 200MB segment, the two files of interest are about 4K bytes. When a large number of segments are being pushed, this results in a significant performance hit.
Instead, it's possible to stream/unpack the segment and extract only the data from the two target files, via something like:
``` java
String uuid = UUID.randomUUID().toString();
File segmentMetadataTarFile = new File(FileUtils.getTempDirectory(), "segmentMetadata-" + uuid + ".tar.gz");
if (segmentMetadataTarFile.exists()) {
FileUtils.forceDelete(segmentMetadataTarFile);
}
GzipCompressorOutputStream gzOut = new GzipCompressorOutputStream(new FileOutputStream(segmentMetadataTarFile));
TarArchiveOutputStream tos = new TarArchiveOutputStream(gzOut);
TarArchiveInputStream tis = new TarArchiveInputStream(new GZIPInputStream(fileSystem.open(segmentUri)));
TarArchiveEntry tarEntry;
while ((tarEntry = tis.getNextTarEntry()) != null) {
System.out.format("%s: %d\n", tarEntry.getName(), tarEntry.getRealSize());
String fullName = tarEntry.getName();
String filename = fullName.substring(fullName.lastIndexOf('/') + 1);
if (tarEntry.isFile() && (filename.equals("metadata.properties") || filename.contentEquals("creation.meta"))) {
TarArchiveEntry ae = new TarArchiveEntry(filename);
ae.setSize(tarEntry.getRealSize());
tos.putArchiveEntry(ae);
IOUtils.copy(tis, tos);
tos.closeArchiveEntry();
}
}
tos.finish();
tos.close();
tis.close();
```
The above is just example code, without exception handling, etc. but it is able to create the required tarball from a source segment.
The real performance win is because the two files of interest occur in the segment tarball before the big files (`columns.psf`, `star_tree_index`, etc) so it would be appropriate to enforce this ordering in the segment writer utility code.
Contributor guide
Research direction
Start at SegmentPushUtils.generateSegmentMetadataFile(PinotFS fileSystem, URI tarFileURI) and inspect the segment writer utility that controls tar entry ordering. Trace fileSystem.open and the metadata entries creation.meta and metadata.properties, then verify the generated tarball contains those files without downloading the full segment before the large entries such as columns.psf and star_tree_index.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- database
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100