apache / apache/parquet-java

Implement async IO for Parquet file reader

Abierto
#2,686 0 comentarios 0 reacciones 0 asignados Ver en GitHub
Component: Java Component: Parquet Priority: Major Type: enhancement
Lenguaje dominante
Java
Estrellas
3.1k
Forks
1.6k
Merge medio
3 d 12 h
PR fusionados (30 d)
33

Descripción

ParquetFileReader's implementation has the following flow (simplified) - 
      - For every column -> Read from storage in 8MB blocks -> Read all uncompressed pages into output queue 
      - From output queues -> (downstream ) decompression + decoding

This flow is serialized, which means that downstream threads are blocked until the data has been read. Because a large part of the time spent is waiting for data from storage, threads are idle and CPU utilization is really low.

There is no reason why this cannot be made asynchronous _and_ parallel. So 

For Column _i_ -> reading one chunk until end, from storage -> intermediate output queue -> read one uncompressed page until end -> output queue -> (downstream ) decompression + decoding

Note that this can be made completely self contained in ParquetFileReader and downstream implementations like Iceberg and Spark will automatically be able to take advantage without code change as long as the ParquetFileReader apis are not changed. 

In past work with async io  [Drill - async page reader ](https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/columnreaders/AsyncPageReader.java) , I have seen 2x-3x improvement in reading speed for Parquet files.

**Reporter**: [Parth Chandra](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=parthc) / @parthchandra
#### Related issues:
- [Improve Parquet IO Performance within cloud datalakes](https://github.com/apache/parquet-java/issues/2912) (is depended upon by)
#### PRs and other links:
- [GitHub Pull Request #968](https://github.com/apache/parquet-java/pull/968)

**Note**: *This issue was originally created as [PARQUET-2149](https://issues.apache.org/jira/browse/PARQUET-2149). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*

Guía de contribución

No hay ninguna guía de contribución indexada para este repositorio

Línea de trabajo

Comienza revisando el flujo serializado actual de ParquetFileReader y el Pull Request #968 mencionado, y luego compara el Drill AsyncPageReader enlazado con el trabajo previo de I/O asíncrono. Se considera terminado cuando las lecturas de almacenamiento y el procesamiento de páginas por columna pueden ejecutarse de forma asíncrona y en paralelo sin cambiar las APIs de ParquetFileReader.

Escrito por el modelo de indexación a partir del texto del issue.

Evaluación

Stack tecnológico
java
Área
data-engineering, performance
Tipo de issue
Nueva funcionalidad
Dificultad
5/5
Tiempo estimado
Más de una semana
Estado de actividad
Estancado
Claridad
Bastante claro
Aptitud para principiantes
25/100

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.