Apache Parquet Java Performance Improvements
- Dominant language
- Java
- Stars
- 3.1k
- Forks
- 1.6k
- Avg merge
- 3d 12h
- Merged PRs (30d)
- 33
Description
Targeted performance optimizations to parquet-java's encoding, decoding, and compression layers. Each PR addresses an independent codec or subsystem, replacing stream-wrapper abstractions with direct ByteBuffer I/O, eliminating per-value allocations, and adding JMH benchmarks to validate the improvements.
All benchmarks: JDK 25.0.3 (Temurin), JMH 1.37, Linux x86_64, 100K values/iteration.
## Pull requests
| PR | Status | Scope | Headline result |
|----|--------|-------|-----------------|
| #3565 | Merged | PLAIN encoding/decoding | Decode 2.6–13x, encode 2–3.8x |
| #3566 | Open | DICTIONARY encoding/decoding | Encode 1.5–100x (high-card / long binary) |
| #3567 | Open | DELTA_BINARY_PACKED, DELTA_LENGTH_BYTE_ARRAY, DELTA_BYTE_ARRAY | Decode +22–31%, encode +3–7% |
| #3568 | Open | RLE/Bit-Packing hybrid codec | Decode +30%, encode +5% |
| #3569 | Merged | BYTE_STREAM_SPLIT | Decode 1.4–5.1x, encode 1.6–6.9x |
| #3570 | Open | Compression (Snappy, ZSTD, LZ4, GZIP, LZO, Brotli) | Isolated decompress +15–25% (Snappy/Zstd/Gzip), Snappy compress up to ~2.9x; enables LZ4/Brotli/LZO without native libs |
| #3571 | Merged | Row group flush buffer lifecycle | Correct resource management (not a perf win) |
PRs 1–6 and 8 are independent and can be reviewed/merged in parallel. Additional PRs for column I/O (par7) and level write batching (par9) will follow once their dependencies (#3565, #3568) land.
## Common optimization patterns
- **InputStream → ByteBuffer**: Direct `ByteBuffer` (LITTLE_ENDIAN) access for `getInt()`/`getLong()`/`getFloat()`/`getDouble()` JVM intrinsics instead of per-byte stream reads.
- **Buffer reuse**: Allocate once, grow lazily — no fresh allocation per page or per run.
- **pack32/unpack32 fast paths**: Batch 32 values per packer/unpacker call (4x fewer invocations).
- **Eliminate intermediate copies**: Write directly to output buffers, bypass stream wrappers.
## Benchmark highlights
### PLAIN (#3565)
| Benchmark | Master | Optimized | Speedup |
|---|---:|---:|---:|
| decodeInt | 425 M ops/s | 5,427 M ops/s | **12.8x** |
| decodeBoolean | 639 | 1,642 | **2.6x** |
| encodeInt | 148 | 559 | **3.8x** |
| encodeBoolean | 850 | 1,692 | **2.0x** |
### DICTIONARY (#3566)
| Benchmark | Master | Optimized | Speedup |
|---|---:|---:|---:|
| encodeBinary LOW len=1000 | 1.5 M ops/s | 148.3 M ops/s | **~100x** |
| encodeBinary LOW len=100 | 13.2 | 107.8 | **8.2x** |
| encodeFlba HIGH len=12 | 6.3 | 15.4 | **2.4x** |
| encodeInt HIGH_CARD | 14.9 | 23.5 | **1.58x** |
### DELTA (#3567)
| Component | Avg improvement |
|---|---:|
| DELTA_BINARY_PACKED decode | **+27%** |
| DELTA_BYTE_ARRAY decode | **+31%** |
| Long delta decode (TIMESTAMP_MILLIS pattern) | **+28%** |
| Encoding (all delta variants) | **+3–7%** |
### RLE (#3568)
| Category | Avg improvement |
|---|---:|
| Direct decoder (packed data) | **+30%** |
| Boolean decode (packed patterns) | **+14%** |
| Encoder | **+5%** |
### BYTE_STREAM_SPLIT (#3569)
| Benchmark | Master | Optimized | Speedup |
|---|---:|---:|---:|
| decodeInt | 203 M ops/s | 1,034 M ops/s | **5.1x** |
| encodeDouble | 53 | 365 | **6.9x** |
| encodeLong | 52 | 356 | **6.9x** |
| encodeInt | 99 | 515 | **5.2x** |
### Compression (#3570)
Isolated `CompressionBenchmark`, ours/master geomean over 16 realistic encoding shapes (>1 = faster):
| Codec | compress | decompress |
|---|---:|---:|
| Snappy | 1.38x | 1.25x |
| Zstd | 1.00x | 1.18x |
| Gzip | 1.00x | 1.15x |
| LZ4_RAW | 0.96x | 1.06x |
Snappy peaks near 2.9x compress / 1.8x decompress on small, highly compressible pages. Compression is ~1–10% of end-to-end write for the fast codecs (up to ~46% for Gzip), so file-level impact scales with codec weight. Also migrates LZO from GPL to Apache 2.0 (aircompressor) and Brotli from abandoned jbrotli to brotli4j (adds aarch64 support); LZ4/Brotli/LZO now work without native Hadoop codecs.
### Row group flush (#3571)
Correct resource management — releases column buffers during flush rather than after. Peak memory unchanged (peak occurs during page compression, not flush). Makes buffers GC-eligible sooner.
## How to run the benchmarks
```bash
# Build the benchmark jar
./mvnw --projects parquet-benchmarks -amd -DskipTests -Denforcer.skip=true clean package
# Quick single-iteration smoke test
./parquet-benchmarks/run.sh all -wi 0 -i 1 -f 1
# Full statistical run
./parquet-benchmarks/run.sh all -wi 5 -i 5 -f 3 -rff /tmp/benchmark-results.json
# Specific benchmark
./parquet-benchmarks/run.sh all "PlainEncodingBenchmark|PlainDecodingBenchmark"
```
## Test validation
Each PR passes the full module test suite:
```bash
./mvnw --projects parquet-column,parquet-common,parquet-hadoop -amd verify
```
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.