apache / apache/arrow

[C++] Lz4HadoopCodec::Compress writes single oversized block incompatible with Hadoop Lz4Decompressor

Open
#49,641 0 comments 0 reactions 1 assignee Claimed by @clee704 View on GitHub
Component: C++
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug

`Lz4HadoopCodec::Compress` writes the entire input as a single Hadoop-framed LZ4 block regardless of size. Hadoop's `Lz4Decompressor` allocates a fixed 256 KiB output buffer per block (`IO_COMPRESSION_CODEC_LZ4_BUFFERSIZE_DEFAULT = 256 * 1024`), so any block whose decompressed size exceeds 256 KiB causes `LZ4Exception` on JVM readers (parquet-mr + Hadoop `BlockDecompressorStream`).

PARQUET-1878 added `Lz4HadoopCodec` but writes one block per page. ARROW-11301 fixed the *reader* for multi-block Hadoop data, but the *writer* was never updated to split large inputs the same way Hadoop's `BlockCompressorStream` does.

### Steps to reproduce

Write a Parquet file with `LZ4_HADOOP` compression containing a dictionary page >256 KiB (e.g. 40K unique INT64 values = 320 KiB), then read it with a JVM-based Parquet reader (parquet-mr + Hadoop).

### Expected behavior

The file should be readable by JVM-based Parquet readers.

### Actual behavior

```
net.jpountz.lz4.LZ4Exception: Error decoding offset 131193 of input buffer
at net.jpountz.lz4.LZ4JNISafeDecompressor.decompress(LZ4JNISafeDecompressor.java:71)
at org.apache.hadoop.io.compress.lz4.Lz4Decompressor.decompressDirectBuf(Lz4Decompressor.java:278)
at org.apache.hadoop.io.compress.BlockDecompressorStream.decompress(BlockDecompressorStream.java:88)
...
```

### Severity

Read failure, not data corruption. The bytes on disk are valid LZ4 — Arrow's own C++ reader handles them fine. The JVM reader throws a hard exception; it does not return wrong data.

### Component(s)

C++

### Related issues

ARROW-9177, PARQUET-1878, ARROW-11301

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.