Apache Parquet Java Performance Improvements
- Langage dominant
- Java
- Étoiles
- 3.1k
- Forks
- 1.6k
- Merge moyen
- 3 j 12 h
- PR mergées (30 j)
- 33
Description
Targeted performance optimizations to parquet-java's encoding, decoding, and compression layers. Each PR addresses an independent codec or subsystem, replacing stream-wrapper abstractions with direct ByteBuffer I/O, eliminating per-value allocations, and adding JMH benchmarks to validate the improvements.
All benchmarks: JDK 25.0.3 (Temurin), JMH 1.37, Linux x86_64, 100K values/iteration.
## Pull requests
| PR | Status | Scope | Headline result |
|----|--------|-------|-----------------|
| #3565 | Merged | PLAIN encoding/decoding | Decode 2.6–13x, encode 2–3.8x |
| #3566 | Open | DICTIONARY encoding/decoding | Encode 1.5–100x (high-card / long binary) |
| #3567 | Open | DELTA_BINARY_PACKED, DELTA_LENGTH_BYTE_ARRAY, DELTA_BYTE_ARRAY | Decode +22–31%, encode +3–7% |
| #3568 | Open | RLE/Bit-Packing hybrid codec | Decode +30%, encode +5% |
| #3569 | Merged | BYTE_STREAM_SPLIT | Decode 1.4–5.1x, encode 1.6–6.9x |
| #3570 | Open | Compression (Snappy, ZSTD, LZ4, GZIP, LZO, Brotli) | Isolated decompress +15–25% (Snappy/Zstd/Gzip), Snappy compress up to ~2.9x; enables LZ4/Brotli/LZO without native libs |
| #3571 | Merged | Row group flush buffer lifecycle | Correct resource management (not a perf win) |
PRs 1–6 and 8 are independent and can be reviewed/merged in parallel. Additional PRs for column I/O (par7) and level write batching (par9) will follow once their dependencies (#3565, #3568) land.
## Common optimization patterns
- **InputStream → ByteBuffer**: Direct `ByteBuffer` (LITTLE_ENDIAN) access for `getInt()`/`getLong()`/`getFloat()`/`getDouble()` JVM intrinsics instead of per-byte stream reads.
- **Buffer reuse**: Allocate once, grow lazily — no fresh allocation per page or per run.
- **pack32/unpack32 fast paths**: Batch 32 values per packer/unpacker call (4x fewer invocations).
- **Eliminate intermediate copies**: Write directly to output buffers, bypass stream wrappers.
## Benchmark highlights
### PLAIN (#3565)
| Benchmark | Master | Optimized | Speedup |
|---|---:|---:|---:|
| decodeInt | 425 M ops/s | 5,427 M ops/s | **12.8x** |
| decodeBoolean | 639 | 1,642 | **2.6x** |
| encodeInt | 148 | 559 | **3.8x** |
| encodeBoolean | 850 | 1,692 | **2.0x** |
### DICTIONARY (#3566)
| Benchmark | Master | Optimized | Speedup |
|---|---:|---:|---:|
| encodeBinary LOW len=1000 | 1.5 M ops/s | 148.3 M ops/s | **~100x** |
| encodeBinary LOW len=100 | 13.2 | 107.8 | **8.2x** |
| encodeFlba HIGH len=12 | 6.3 | 15.4 | **2.4x** |
| encodeInt HIGH_CARD | 14.9 | 23.5 | **1.58x** |
### DELTA (#3567)
| Component | Avg improvement |
|---|---:|
| DELTA_BINARY_PACKED decode | **+27%** |
| DELTA_BYTE_ARRAY decode | **+31%** |
| Long delta decode (TIMESTAMP_MILLIS pattern) | **+28%** |
| Encoding (all delta variants) | **+3–7%** |
### RLE (#3568)
| Category | Avg improvement |
|---|---:|
| Direct decoder (packed data) | **+30%** |
| Boolean decode (packed patterns) | **+14%** |
| Encoder | **+5%** |
### BYTE_STREAM_SPLIT (#3569)
| Benchmark | Master | Optimized | Speedup |
|---|---:|---:|---:|
| decodeInt | 203 M ops/s | 1,034 M ops/s | **5.1x** |
| encodeDouble | 53 | 365 | **6.9x** |
| encodeLong | 52 | 356 | **6.9x** |
| encodeInt | 99 | 515 | **5.2x** |
### Compression (#3570)
Isolated `CompressionBenchmark`, ours/master geomean over 16 realistic encoding shapes (>1 = faster):
| Codec | compress | decompress |
|---|---:|---:|
| Snappy | 1.38x | 1.25x |
| Zstd | 1.00x | 1.18x |
| Gzip | 1.00x | 1.15x |
| LZ4_RAW | 0.96x | 1.06x |
Snappy peaks near 2.9x compress / 1.8x decompress on small, highly compressible pages. Compression is ~1–10% of end-to-end write for the fast codecs (up to ~46% for Gzip), so file-level impact scales with codec weight. Also migrates LZO from GPL to Apache 2.0 (aircompressor) and Brotli from abandoned jbrotli to brotli4j (adds aarch64 support); LZ4/Brotli/LZO now work without native Hadoop codecs.
### Row group flush (#3571)
Correct resource management — releases column buffers during flush rather than after. Peak memory unchanged (peak occurs during page compression, not flush). Makes buffers GC-eligible sooner.
## How to run the benchmarks
```bash
# Build the benchmark jar
./mvnw --projects parquet-benchmarks -amd -DskipTests -Denforcer.skip=true clean package
# Quick single-iteration smoke test
./parquet-benchmarks/run.sh all -wi 0 -i 1 -f 1
# Full statistical run
./parquet-benchmarks/run.sh all -wi 5 -i 5 -f 3 -rff /tmp/benchmark-results.json
# Specific benchmark
./parquet-benchmarks/run.sh all "PlainEncodingBenchmark|PlainDecodingBenchmark"
```
## Test validation
Each PR passes the full module test suite:
```bash
./mvnw --projects parquet-column,parquet-common,parquet-hadoop -amd verify
```
Guide de contribution
Aucun guide de contribution indexé pour ce dépôt
Piste de recherche
Commencez par examiner les PR ouvertes #3566, #3567, #3568 et #3570, puisque le travail d’optimisation a été réparti entre des sous-systèmes indépendants. Exécutez les commandes de benchmark indiquées dans l’issue pour la zone de codec ou de compression concernée, puis validez la fin du travail avec la suite de tests des modules parquet-column, parquet-common et parquet-hadoop.
Rédigé par le modèle d'indexation à partir du texte de l'issue.
Évaluation
- Stack technique
- java
- Domaine
- data-engineering, performance
- Type d'issue
- Refactorisation
- Difficulté
- 5/5
- Temps estimé
- Plus d'une semaine
- Activité
- À l'abandon
- Clarté
- À clarifier
- Accessibilité débutants
- 20/100