Avro adapter: setPosition doesn't work with BaseVariableWidthVectors
- 主要語言
- Java
- 星號
- 94
- 分支
- 152
- 平均合併
- 3 天 16 小時
- 30 天內合併 PR
- 11
描述
### Describe the bug, including details regarding any error messages, version, and platform.
`BaseVariableWidthVector`-based vectors (`VarBinaryVector`, `VarCharVector`) expect the data to be written sequentially. This means `Consumer.setPosition` is not safe to use. Imagine the following scenario:
- write index 0
- write index 2
- write index 1
- write index 3
On the last step, `BaseVariableWidthVector.fillHoles` erases the data written for the index 2:
```java
protected final void fillHoles(int index) {
for (int i = lastSet + 1; i < index; i++) {
setBytes(i, emptyByteArray, 0, emptyByteArray.length);
}
lastSet = index - 1;
}
```
Here, index would be 3, lastSet + 1 would be 2 (1 + 1), so `setBytes(2, emptyByteArray, ...)` is called.
Not sure what the right solution is here, but at a minimum, the library shouldn't allow calling `setPosition` for Consumers using `BaseVariableWidthVector` vectors.
貢獻指南
研究方向
首先,追蹤 Avro 配接器在使用以 BaseVariableWidthVector 為基礎的 VarBinaryVector 和 VarCharVector 實例時 Consumer.setPosition 的行為。重現寫入序列 0、2、1、3 並新增回歸測試;當定位不再清除先前寫入的資料,或不再阻止對這些向量進行定位時,即表示完成。
由索引模型根據 Issue 內容生成。
評估
- 技術堆疊
- java
- 領域
- data
- Issue 類型
- 缺陷
- 難度
- 4/5
- 預估耗時
- 3-5 天
- 活躍度
- 停滯
- 描述清晰度
- 需要釐清
- 新手友好度
- 35/100