apache / apache/parquet-java

Benchmark the assembly of thrift objects, and possibly create a more efficient ReplayingTProtocol

Abierto
#1,391 2 comentarios 0 reacciones 0 asignados Ver en GitHub
Component: Java Component: Parquet Priority: Minor Type: enhancement
Lenguaje dominante
Java
Estrellas
3.1k
Forks
1.6k
Merge medio
3 d 12 h
PR fusionados (30 d)
33

Descripción

The current implementation of parquet thrift creates an instance of TProtocol for each value of each record and builds a stack of these events, which are then replayed back to the TBase.

I'd be curious to benchmark this, and if it's slow, try building a "ReplayingTProtocol" that instead of having a stack of TProtocol instances, contains a primitive array of each type. As events are fed into this replaying TProtocol, it would just add these primitives to its buffers, and then the TBase would drain them. This would effectively let us stream the values into the TBase without making an object allocation for each value.

The buffers could be set to a certain size, and if they fill up (which they sholdn't in most cases), the TBase could begin draining the protocol until it is empty again, at which point the TProtocol can block the TBase from draining further while the parque record assembly feeds it more events.

This is all moot if it turns out not to be bottleneck though :)

**Reporter**: [Alex Levenson](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=alexlevenson) / @isnotinvain

**Note**: *This issue was originally created as [PARQUET-33](https://issues.apache.org/jira/browse/PARQUET-33). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*

Guía de contribución

No hay ninguna guía de contribución indexada para este repositorio

Línea de trabajo

Comienza localizando la ruta actual de ensamblaje de registros de parquet thrift que crea un TProtocol por valor y ejecuta un benchmark antes de diseñar cambios. Compara la asignación y el rendimiento con la reproducción existente de event-stack; se considera terminado cuando el benchmark respalda una decisión clara sobre si está justificado un ReplayingTProtocol, con el alcance y las pruebas documentados.

Escrito por el modelo de indexación a partir del texto del issue.

Evaluación

Stack tecnológico
java
Área
performance
Tipo de issue
Refactorización
Dificultad
5/5
Tiempo estimado
Más de una semana
Estado de actividad
Estancado
Claridad
Necesita aclaración
Aptitud para principiantes
18/100

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.