apache / apache/parquet-java

Improve SnappyCompressor buffer expansion performance

Offen
#1,647 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
Component: Java Component: Parquet Priority: Minor Type: enhancement
Vorherrschende Sprache
Java
Sterne
3.1k
Forks
1.6k
Ø Merge
3 T. 12 Std.
Gemergte PRs (30 T.)
33

Beschreibung

The existing implementation of SnappyCompressor will only allocate enough bytes for the buffer passed into setInput().  This leads to suboptimal performance when there are patterns of writes that cause repeated buffer expansions.  In the worst case it must copy the entire buffer for every single invocation of setInput()

Instead of allocating a buffer of size current + write length,  there should be an expansion strategy that reduces the amount of copying required.

**Reporter**: [Andrew Baranec](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=abaranec)

**Note**: *This issue was originally created as [PARQUET-2184](https://issues.apache.org/jira/browse/PARQUET-2184). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*

Beitragsleitfaden

Für dieses Repository ist kein Beitragsleitfaden indexiert

Rechercherichtung

Locate SnappyCompressor and its setInput() implementation, then inspect how buffer allocation and copying behave across repeated writes. The work is done when buffer expansion avoids copying the entire buffer on every invocation while preserving the compressor's existing input behavior.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Bewertung

Tech-Stack
java
Bereich
performance
Issue-Typ
Refactoring
Schwierigkeit
3/5
Geschätzter Aufwand
1-2 Tage
Aktivitätsstatus
Veraltet
Klarheit
Größtenteils klar
Anfängerfreundlichkeit
45/100

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.