feat: reduce temp copy when decompressing lz4 frame
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 2d 18h
- Merged PRs (30d)
- 169
Description
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
We hit this while using **fluss-rust column projection**: when projecting more fields, LZ4 decompression cost grows rapidly. In practice, with **174/775 projected columns**, consumption capacity drops sharply and d
ecompression becomes the dominant bottleneck. This points to the LZ4 IPC decode path (arrow-ipc) as the critical limiter for high-cardinality projections.
Concrete observations (same workload):
- arrow-rs (frame decoder): `avg_decode_ms ≈ 26.4`, `decode_util ≈ 99.8%`
- arrow-java (decoder=arrow): `avg_decode_ms ≈ 0.64`, `decode_util ≈ 11.6%`
This strongly suggests the streaming FrameDecoder path (state machine + Read trait + buffer resize/zero-init) adds significant overhead beyond core LZ4 block decompression.
**Describe the solution you'd like**
Add a direct LZ4 frame parsing path in `arrow-ipc` that avoids the streaming FrameDecoder overhead while keeping the same correctness guarantees:
1) **Header parsing & validation**
- magic / version / reserved bits / block size / block independence
- header checksum (XXH32 >> 8)
- content size (if present)
- dictionary id: return error (same behavior as current path)
2) **Per-block handling**
- read block size + incompressible flag
- block checksum verification if enabled
- compressed block → `lz4_flex::block::decompress_into`
- incompressible block → direct copy
3) **Content checksum**
- compute XXH32 on decompressed output
- verify at end if enabled
4) **Output**
- pre-allocate `decompressed_size`, write directly into output
- avoid `read_exact + vec_resize_and_get_mut` / buffer re-init
Compatibility goal: no API changes; internal implementation swap. Keep current semantics (dict id unsupported, checksum and size checks preserved).
Expected outcome (observed in local prototype):
- arrow-rs (optimized): `avg_decode_ms ≈ 0.47`, `decode_util ≈ 8.6%`
**Describe alternatives you've considered**
**Additional context**
Top bottlenecks & mitigations:
1) **LZ4 block core decompression**
- Dominant CPU cost; cannot be removed.
- Optimization removes surrounding overhead so this becomes the only hot path.
2) **FrameDecoder state machine + Read trait path**
- Significant overhead due to streaming logic and state transitions.
- Direct frame parsing removes it entirely.
3) **Buffer management / extra copies**
- `read_exact + vec_resize_and_get_mut` causes resize/zero-init overhead.
- Pre-allocated output + direct block writes reduce allocations.
Contributor guide
Research direction
Start in arrow-ipc by locating the streaming FrameDecoder path and the existing lz4_flex block decompression calls. Trace its header, block, checksum, dictionary-id, and size validation behavior before assessing a direct parsing path. Done means the internal path avoids the extra buffer copy while preserving the stated checks and semantics without changing the API.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- performance
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 47/100