apache / apache/parquet-java

Improve SnappyCompressor buffer expansion performance

Abierto
#1,647 0 comentarios 0 reacciones 0 asignados Ver en GitHub
Component: Java Component: Parquet Priority: Minor Type: enhancement
Lenguaje dominante
Java
Estrellas
3.1k
Forks
1.6k
Merge medio
3 d 12 h
PR fusionados (30 d)
33

Descripción

The existing implementation of SnappyCompressor will only allocate enough bytes for the buffer passed into setInput().  This leads to suboptimal performance when there are patterns of writes that cause repeated buffer expansions.  In the worst case it must copy the entire buffer for every single invocation of setInput()

Instead of allocating a buffer of size current + write length,  there should be an expansion strategy that reduces the amount of copying required.

**Reporter**: [Andrew Baranec](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=abaranec)

**Note**: *This issue was originally created as [PARQUET-2184](https://issues.apache.org/jira/browse/PARQUET-2184). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*

Guía de contribución

No hay ninguna guía de contribución indexada para este repositorio

Línea de trabajo

Locate SnappyCompressor and its setInput() implementation, then inspect how buffer allocation and copying behave across repeated writes. The work is done when buffer expansion avoids copying the entire buffer on every invocation while preserving the compressor's existing input behavior.

Escrito por el modelo de indexación a partir del texto del issue.

Evaluación

Stack tecnológico
java
Área
performance
Tipo de issue
Refactorización
Dificultad
3/5
Tiempo estimado
1-2 días
Estado de actividad
Estancado
Claridad
Bastante claro
Aptitud para principiantes
45/100

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.