[Java] DictionaryProvider leaks memory while adding dictionaries with duplicate encoding
- 主要語言
- Java
- 星號
- 94
- 分支
- 152
- 平均合併
- 3 天 16 小時
- 30 天內合併 PR
- 11
描述
DictionaryProvider leaks memory while adding dictionaries with duplicate encoding. Is this expected? Should the provider release the memory of the existing dictionary vector if it accepts another one with same encoding id ?
Sample code:
```java
"dictionaryProvider" should " not leak memory while adding dictionaries with duplicate encoding" in {
val allocator: RootAllocator = new RootAllocator()
val vector: ListVector = ListVector.empty("vector", allocator)
val dictionaryVector1: ListVector = ListVector.empty("dict1", allocator)
val dictionaryVector2: ListVector = ListVector.empty("dict2", allocator)
val writer1: UnionListWriter = vector.getWriter
writer1.allocate
writer1.setValueCount(1)
val dictWriter1: UnionListWriter = dictionaryVector1.getWriter
dictWriter1.allocate
dictWriter1.setValueCount(1)
val dictWriter2: UnionListWriter = dictionaryVector2.getWriter
dictWriter2.allocate
dictWriter2.setValueCount(1)
val dictionary1: Dictionary = new Dictionary(dictionaryVector1, new DictionaryEncoding(1L, false, None.orNull))
val dictionary2: Dictionary = new Dictionary(dictionaryVector2, new DictionaryEncoding(1L, false, None.orNull))
val provider = new DictionaryProvider.MapDictionaryProvider
provider.put(dictionary1)
provider.put(dictionary2)
vector.clear()
provider.getDictionaryIds.asScala.map(id => provider.lookup(id).getVector.clear())
allocator.getAllocatedMemory shouldBe 0
}
```
**Reporter**: [Vimal Varghese](https://issues.apache.org/jira/browse/ARROW-16920)
**Note**: *This issue was originally created as [ARROW-16920](https://issues.apache.org/jira/browse/ARROW-16920). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*
貢獻指南
研究方向
從 DictionaryProvider.MapDictionaryProvider 開始,執行提供的重現步驟:加入兩個 encoding ID 為 1 的字典,清空 vector,並檢查 allocator 記憶體。追蹤插入第二個字典時第一個字典發生了什麼;完成標準是重複 encoding 的情況釋放所有已配置的記憶體,且最後的配置檢查達到零。
由索引模型根據 Issue 內容生成。
評估
- 技術堆疊
- java
- 領域
- data
- Issue 類型
- 缺陷
- 難度
- 3/5
- 預估耗時
- 1-2 天
- 活躍度
- 停滯
- 描述清晰度
- 基本清楚
- 新手友好度
- 35/100