apache / apache/parquet-java

Apache Parquet Java Performance Improvements

Open
#3,530 5 comments 7 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
3.1k
Forks
1.6k
Avg merge
3d 12h
Merged PRs (30d)
33

Description

Targeted performance optimizations to parquet-java's encoding, decoding, and compression layers. Each PR addresses an independent codec or subsystem, replacing stream-wrapper abstractions with direct ByteBuffer I/O, eliminating per-value allocations, and adding JMH benchmarks to validate the improvements.

All benchmarks: JDK 25.0.3 (Temurin), JMH 1.37, Linux x86_64, 100K values/iteration.

## Pull requests

| PR | Status | Scope | Headline result |
|----|--------|-------|-----------------|
| #3565 | Merged | PLAIN encoding/decoding | Decode 2.6–13x, encode 2–3.8x |
| #3566 | Open | DICTIONARY encoding/decoding | Encode 1.5–100x (high-card / long binary) |
| #3567 | Open | DELTA_BINARY_PACKED, DELTA_LENGTH_BYTE_ARRAY, DELTA_BYTE_ARRAY | Decode +22–31%, encode +3–7% |
| #3568 | Open | RLE/Bit-Packing hybrid codec | Decode +30%, encode +5% |
| #3569 | Merged | BYTE_STREAM_SPLIT | Decode 1.4–5.1x, encode 1.6–6.9x |
| #3570 | Open | Compression (Snappy, ZSTD, LZ4, GZIP, LZO, Brotli) | Isolated decompress +15–25% (Snappy/Zstd/Gzip), Snappy compress up to ~2.9x; enables LZ4/Brotli/LZO without native libs |
| #3571 | Merged | Row group flush buffer lifecycle | Correct resource management (not a perf win) |

PRs 1–6 and 8 are independent and can be reviewed/merged in parallel. Additional PRs for column I/O (par7) and level write batching (par9) will follow once their dependencies (#3565, #3568) land.

## Common optimization patterns

- **InputStream → ByteBuffer**: Direct `ByteBuffer` (LITTLE_ENDIAN) access for `getInt()`/`getLong()`/`getFloat()`/`getDouble()` JVM intrinsics instead of per-byte stream reads.
- **Buffer reuse**: Allocate once, grow lazily — no fresh allocation per page or per run.
- **pack32/unpack32 fast paths**: Batch 32 values per packer/unpacker call (4x fewer invocations).
- **Eliminate intermediate copies**: Write directly to output buffers, bypass stream wrappers.

## Benchmark highlights

### PLAIN (#3565)

| Benchmark | Master | Optimized | Speedup |
|---|---:|---:|---:|
| decodeInt | 425 M ops/s | 5,427 M ops/s | **12.8x** |
| decodeBoolean | 639 | 1,642 | **2.6x** |
| encodeInt | 148 | 559 | **3.8x** |
| encodeBoolean | 850 | 1,692 | **2.0x** |

### DICTIONARY (#3566)

| Benchmark | Master | Optimized | Speedup |
|---|---:|---:|---:|
| encodeBinary LOW len=1000 | 1.5 M ops/s | 148.3 M ops/s | **~100x** |
| encodeBinary LOW len=100 | 13.2 | 107.8 | **8.2x** |
| encodeFlba HIGH len=12 | 6.3 | 15.4 | **2.4x** |
| encodeInt HIGH_CARD | 14.9 | 23.5 | **1.58x** |

### DELTA (#3567)

| Component | Avg improvement |
|---|---:|
| DELTA_BINARY_PACKED decode | **+27%** |
| DELTA_BYTE_ARRAY decode | **+31%** |
| Long delta decode (TIMESTAMP_MILLIS pattern) | **+28%** |
| Encoding (all delta variants) | **+3–7%** |

### RLE (#3568)

| Category | Avg improvement |
|---|---:|
| Direct decoder (packed data) | **+30%** |
| Boolean decode (packed patterns) | **+14%** |
| Encoder | **+5%** |

### BYTE_STREAM_SPLIT (#3569)

| Benchmark | Master | Optimized | Speedup |
|---|---:|---:|---:|
| decodeInt | 203 M ops/s | 1,034 M ops/s | **5.1x** |
| encodeDouble | 53 | 365 | **6.9x** |
| encodeLong | 52 | 356 | **6.9x** |
| encodeInt | 99 | 515 | **5.2x** |

### Compression (#3570)

Isolated `CompressionBenchmark`, ours/master geomean over 16 realistic encoding shapes (>1 = faster):

| Codec | compress | decompress |
|---|---:|---:|
| Snappy | 1.38x | 1.25x |
| Zstd | 1.00x | 1.18x |
| Gzip | 1.00x | 1.15x |
| LZ4_RAW | 0.96x | 1.06x |

Snappy peaks near 2.9x compress / 1.8x decompress on small, highly compressible pages. Compression is ~1–10% of end-to-end write for the fast codecs (up to ~46% for Gzip), so file-level impact scales with codec weight. Also migrates LZO from GPL to Apache 2.0 (aircompressor) and Brotli from abandoned jbrotli to brotli4j (adds aarch64 support); LZ4/Brotli/LZO now work without native Hadoop codecs.

### Row group flush (#3571)

Correct resource management — releases column buffers during flush rather than after. Peak memory unchanged (peak occurs during page compression, not flush). Makes buffers GC-eligible sooner.

## How to run the benchmarks

```bash
# Build the benchmark jar
./mvnw --projects parquet-benchmarks -amd -DskipTests -Denforcer.skip=true clean package

# Quick single-iteration smoke test
./parquet-benchmarks/run.sh all -wi 0 -i 1 -f 1

# Full statistical run
./parquet-benchmarks/run.sh all -wi 5 -i 5 -f 3 -rff /tmp/benchmark-results.json

# Specific benchmark
./parquet-benchmarks/run.sh all "PlainEncodingBenchmark|PlainDecodingBenchmark"
```

## Test validation

Each PR passes the full module test suite:

```bash
./mvnw --projects parquet-column,parquet-common,parquet-hadoop -amd verify
```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.