apache / apache/parquet-java

Implement async IO for Parquet file reader

Ouverte
#2,686 0 commentaires 0 réactions 0 personnes assignées Voir sur GitHub
Component: Java Component: Parquet Priority: Major Type: enhancement
Langage dominant
Java
Étoiles
3.1k
Forks
1.6k
Merge moyen
3 j 12 h
PR mergées (30 j)
33

Description

ParquetFileReader's implementation has the following flow (simplified) - 
      - For every column -> Read from storage in 8MB blocks -> Read all uncompressed pages into output queue 
      - From output queues -> (downstream ) decompression + decoding

This flow is serialized, which means that downstream threads are blocked until the data has been read. Because a large part of the time spent is waiting for data from storage, threads are idle and CPU utilization is really low.

There is no reason why this cannot be made asynchronous _and_ parallel. So 

For Column _i_ -> reading one chunk until end, from storage -> intermediate output queue -> read one uncompressed page until end -> output queue -> (downstream ) decompression + decoding

Note that this can be made completely self contained in ParquetFileReader and downstream implementations like Iceberg and Spark will automatically be able to take advantage without code change as long as the ParquetFileReader apis are not changed. 

In past work with async io  [Drill - async page reader ](https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/columnreaders/AsyncPageReader.java) , I have seen 2x-3x improvement in reading speed for Parquet files.

**Reporter**: [Parth Chandra](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=parthc) / @parthchandra
#### Related issues:
- [Improve Parquet IO Performance within cloud datalakes](https://github.com/apache/parquet-java/issues/2912) (is depended upon by)
#### PRs and other links:
- [GitHub Pull Request #968](https://github.com/apache/parquet-java/pull/968)

**Note**: *This issue was originally created as [PARQUET-2149](https://issues.apache.org/jira/browse/PARQUET-2149). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*

Guide de contribution

Aucun guide de contribution indexé pour ce dépôt

Piste de recherche

Commencez par examiner le flux sérialisé actuel de ParquetFileReader et le Pull Request #968 référencé, puis comparez le Drill AsyncPageReader lié avec les travaux précédents sur les I/O asynchrones. Le travail est terminé lorsque les lectures du stockage et le traitement des pages par colonne peuvent s'exécuter de manière asynchrone et en parallèle sans modifier les APIs de ParquetFileReader.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Évaluation

Stack technique
java
Domaine
data-engineering, performance
Type d'issue
Fonctionnalité
Difficulté
5/5
Temps estimé
Plus d'une semaine
Activité
À l'abandon
Clarté
Plutôt claire
Accessibilité débutants
25/100

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.