apache / apache/parquet-java

Use airlift non-native implementations for GZIP, LZ0 and LZ4 codecs

Open
#2,359 1 comment 0 reactions 0 assignees View on GitHub
Component: Parquet Priority: Major Type: enhancement
Dominant language
Java
Stars
3.1k
Forks
1.6k
Avg merge
3d 12h
Merged PRs (30d)
33

Description

@rdblue  pointed me to  which provides non-native implementations of compression codecs. It claims to be much faster than native wrappers that parquet uses. This Jira is to track the work needed for exploring using these codecs, getting benchmark results and making changes including not needing to pool compressors and decompressors anymore. Note that this doesn't include SNAPPY since Parquet already has its own non-hadoopy implementation for it. 

**Reporter**: [Samarth Jain](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=samarthjain) / @samarthjain
**Assignee**: [Samarth Jain](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=samarthjain) / @samarthjain
#### PRs and other links:
- [GitHub Pull Request #671](https://github.com/apache/parquet-mr/pull/671)

**Note**: *This issue was originally created as [PARQUET-1643](https://issues.apache.org/jira/browse/PARQUET-1643). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reviewing airlift/aircompressor and the work recorded in pull request #671. Investigate the GZIP, LZ0, and LZ4 codec changes and benchmark results, including whether compressor and decompressor pooling is still needed. Done means the exploration, benchmarks, and related Parquet changes are complete and documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data, performance
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
15/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.