apache / apache/arrow-java

Java compression codecs do not release compressed ArrowBuf in decompress, causing allocator leaks

未关闭
#1,037 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
Type: bug
主要语言
Java
星标
94
派生
152
平均合并
3 天 16 小时
30 天内合并 PR
11

描述

### bug
When using Arrow Java’s compression codecs (e.g. ZSTD) from downstream projects, we observed that an OutOfMemoryError: Direct buffer memory during decompression can leave the associated BufferAllocator in a leaked state (its allocated memory never returns to zero after the failing operation).

In our case this surfaced in Apache Fluss (see [apache/fluss#2646](https://github.com/apache/fluss/issues/2646)), but after investigation the root cause appears to be in Arrow Java’s compression codec implementation:

Image

### version
18.3.0

### Solve
In the original Arrow implementation, the decompression loop runs outside the try-finally block that guards loadFieldBuffers. This means if decompression succeeds for the first N buffers of a field but fails on the (N+1)-th buffer, the already-decompressed buffers in ownBuffers are never closed, leaking Direct Memory.

To fix it, move the decompression loop inside the try block so that the finally clause always closes every buffer in ownBuffers, regardless of whether the load succeeds or fails:
* Success path: loadFieldBuffers retains each buffer (ref count +1), then the finally close decrements it back (ref count -1). The field vector still holds the buffer.
* Error path: The finally close decrements each already-decompressed buffer's ref count to 0, immediately freeing the Direct Memory.

贡献指南

打开贡献指南

调研方向

定位包含 loadFieldBuffers、解压循环和 ownBuffers 的压缩 codec 解压路径。将循环移到现有 try-finally 清理逻辑之下,然后验证在较早的缓冲区完成解压后发生失败时,BufferAllocator 的已分配内存为零,而成功路径仍会在字段向量中保留缓冲区。

由索引模型根据 Issue 内容生成。

评估

技术栈
java
领域
data
Issue 类型
缺陷
难度
3/5
预计耗时
1-2 天
活跃度
停滞
描述清晰度
基本清楚
新手友好度
55/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。