apache / apache/parquet-java

Improve `RunLengthBitPackingHybridDecoder.readNext` to avoid per-call buffer allocation and `DataInputStream` wrapping

未关闭 适合新手
#3,466 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
Type: enhancement
主要语言
Java
星标
3.1k
派生
1.6k
平均合并
3 天 12 小时
30 天内合并 PR
33

描述

### Describe the enhancement requested

`RunLengthBitPackingHybridDecoder.readNext()` allocates a new `int[]` and `byte[]` on every PACKED-mode call. In workloads that decode many bit-packed runs (definition levels, repetition levels, RLE-encoded integers), these allocations dominate the read-side allocation profile. The upstream code even acknowledges this with a `// TODO: reuse a buffer` comment.

### Problem 1: per-call buffer allocation

[Lines 94–95](https://github.com/apache/parquet-java/blob/4c8f4d4b/parquet-column/src/main/java/org/apache/parquet/column/values/rle/RunLengthBitPackingHybridDecoder.java#L94-L95) allocate fresh arrays on every PACKED-mode `readNext()`:

```java
currentBuffer = new int[currentCount]; // TODO: reuse a buffer
byte[] bytes = new byte[numGroups * bitWidth];
```

`currentCount` is always `numGroups * 8`, and `numGroups` is typically small (1–16 groups = 8–128 values per run). These allocations are individually modest but occur thousands of times per column chunk — once per bit-packed run. In a 180M-row merge with multiple integer/boolean columns, the cumulative allocation is substantial.

Since `currentCount` varies between runs (different `numGroups` values), the fix retains the field-level `int[]` and a new field-level `byte[]`, growing them only when the next run requires a larger buffer.

### Problem 2: per-call DataInputStream wrapping

[Line 98](https://github.com/apache/parquet-java/blob/4c8f4d4b/parquet-column/src/main/java/org/apache/parquet/column/values/rle/RunLengthBitPackingHybridDecoder.java#L98) creates a `new DataInputStream(in)` on every PACKED-mode call:

```java
new DataInputStream(in).readFully(bytes, 0, bytesToRead);
```

This allocates a `DataInputStream` wrapper object per call just to access `readFully()`. A private `readFully()` method on the decoder itself eliminates this allocation and the virtual dispatch through the wrapper.

### Component(s)

Core

贡献指南

这个仓库没有索引到贡献指南

调研方向

从 parquet-column/src/main/java/org/apache/parquet/column/values/rle/RunLengthBitPackingHybridDecoder.java 开始,重点查看第 94–98 行附近 readNext() 中的 PACKED-mode 路径以及现有的 TODO。跟踪缓冲区和输入读取的使用方式,然后运行相关的 decoder 测试。当重复调用 PACKED-mode 时能够复用大小足够的缓冲区,而不会为每次调用创建 DataInputStream 包装器,同时保持解码行为不变,即表示完成。

由索引模型根据 Issue 内容生成。

评估

技术栈
java
领域
data
Issue 类型
重构
难度
2/5
预计耗时
1-3 小时
活跃度
冷清
描述清晰度
描述清楚
新手友好度
74/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。