apache / apache/arrow-java

Java compression codecs do not release compressed ArrowBuf in decompress, causing allocator leaks

未關閉
#1,037 0 則留言 0 個 reaction 已指派 0 人 在 GitHub 檢視
Type: bug
主要語言
Java
星號
94
分支
152
平均合併
3 天 16 小時
30 天內合併 PR
11

描述

### bug
When using Arrow Java’s compression codecs (e.g. ZSTD) from downstream projects, we observed that an OutOfMemoryError: Direct buffer memory during decompression can leave the associated BufferAllocator in a leaked state (its allocated memory never returns to zero after the failing operation).

In our case this surfaced in Apache Fluss (see [apache/fluss#2646](https://github.com/apache/fluss/issues/2646)), but after investigation the root cause appears to be in Arrow Java’s compression codec implementation:

Image

### version
18.3.0

### Solve
In the original Arrow implementation, the decompression loop runs outside the try-finally block that guards loadFieldBuffers. This means if decompression succeeds for the first N buffers of a field but fails on the (N+1)-th buffer, the already-decompressed buffers in ownBuffers are never closed, leaking Direct Memory.

To fix it, move the decompression loop inside the try block so that the finally clause always closes every buffer in ownBuffers, regardless of whether the load succeeds or fails:
* Success path: loadFieldBuffers retains each buffer (ref count +1), then the finally close decrements it back (ref count -1). The field vector still holds the buffer.
* Error path: The finally close decrements each already-decompressed buffer's ref count to 0, immediately freeing the Direct Memory.

貢獻指南

開啟貢獻指南

研究方向

定位包含 loadFieldBuffers、解壓縮迴圈和 ownBuffers 的壓縮 codec 解壓縮路徑。將迴圈移到現有 try-finally 清理邏輯之下,然後驗證在較早的緩衝區完成解壓縮後發生失敗時,BufferAllocator 的已配置記憶體為零,而成功路徑仍會在欄位向量中保留緩衝區。

由索引模型根據 Issue 內容生成。

評估

技術堆疊
java
領域
data
Issue 類型
缺陷
難度
3/5
預估耗時
1-2 天
活躍度
停滯
描述清晰度
基本清楚
新手友好度
55/100

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。