apache / apache/parquet-java

Add support for Parquet Input Stream optimisations.

Open
#3,386 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
3.1k
Forks
1.6k
Avg merge
3d 12h
Merged PRs (30d)
33

Description

This issue tracks adding of a new module [parquet-io] with the goal of adding IO optimisations to the parquet-java repository.

The goal is to have a single place where the following optimisations can be implemented:

* Vectored Reads
* Reading the tail of a file in one request rather than multiple small requests (Avoid the parquet footer dance, multiple requests for the pageIndex)
* Small Parquet files are read in a single request
* Sequential prefetching

[Parquet Java Input Stream Optimisations](https://docs.google.com/document/d/1Xdlh23tmCs-KvzHhY2RuwFYmc3xntUKcmwb8yxEl78Y/edit?usp=sharing): Doc explains the features this will implement.

[Analytics Accelerator for S3](https://docs.google.com/document/d/13shy0RWotwfWC_qQksb95PXdi-vSUCKQyDzjoExQEN0/edit?tab=t.0#heading=h.3lc3p7s26rnw): Doc explains IO optimisations made in the analytics accelerator library.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading the Parquet Java Input Stream Optimisations document and the Analytics Accelerator for S3 document linked in the issue. Then define the scope of the proposed parquet-io module around vectored reads, tail and small-file reads, page-index access, and sequential prefetching. Done would require an agreed implementation plan and support for the listed optimisations.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.