apache / apache/parquet-java

Improve SnappyCompressor buffer expansion performance

Open
#1,647 0 comments 0 reactions 0 assignees View on GitHub
Component: Java Component: Parquet Priority: Minor Type: enhancement
Dominant language
Java
Stars
3.1k
Forks
1.6k
Avg merge
3d 12h
Merged PRs (30d)
33

Description

The existing implementation of SnappyCompressor will only allocate enough bytes for the buffer passed into setInput().  This leads to suboptimal performance when there are patterns of writes that cause repeated buffer expansions.  In the worst case it must copy the entire buffer for every single invocation of setInput()

Instead of allocating a buffer of size current + write length,  there should be an expansion strategy that reduces the amount of copying required.

**Reporter**: [Andrew Baranec](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=abaranec)

**Note**: *This issue was originally created as [PARQUET-2184](https://issues.apache.org/jira/browse/PARQUET-2184). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.