[Java] Unable to update VectorSchemaRoot field vectors.
- 主要語言
- Java
- 星號
- 94
- 分支
- 152
- 平均合併
- 3 天 16 小時
- 30 天內合併 PR
- 11
描述
### Describe the usage question you have. Please include as many useful details as possible.
Hi community,
I have a trying to modify data buffers of string column.
if i print the buffer readable bytes after ` buffers.set(index, compressedBuf); `, i see the right value.
```
List buffers = fieldVector.getFieldBuffers();
int index = buffers.size() - 1;
ArrowBuf compressedBuf = null;
ArrowBuf originalBuf = buffers.get(index);
ByteBuffer originalBuffer = originalBuf.nioBuffer(0, (int) originalBuf.readableBytes());
// Compress the buffer using LZ4
byte[] originalBytes = new byte[originalBuffer.remaining()];
System.out.println("This is buffer id: "+ index + " originalbytes: "+ originalBuf.readableBytes());
originalBuffer.get(originalBytes);
int maxCompressedLength = compressor.maxCompressedLength(originalBytes.length);
byte[] compressedBytes = new byte[maxCompressedLength];
int compressedLength = compressor.compress(originalBytes, 0, originalBytes.length, compressedBytes, 0, maxCompressedLength);
// Allocate a new ArrowBuf for the compressed data
compressedBuf = allocator.buffer(compressedLength);
compressedBuf.writeBytes(compressedBytes, 0, compressedLength);
compressedBuf.writerIndex(compressedLength);
System.out.println("This is buffer id: "+ index+ " compressedbytes: "+ compressedBuf.readableBytes());
buffers.set(index, compressedBuf); // Replace with compressed buffer
```
When i update my fieldvector.
```
int nullCount = fieldVector.getNullCount(); // Get the null count
ArrowFieldNode fieldNode = new ArrowFieldNode(fieldVectorValueCount, nullCount);
ArrowBuf validityBuffer = fieldVector.getValidityBuffer(); // Assuming it exists
ArrowBuf offsetBuffer = fieldVector.getOffsetBuffer(); // Assuming it exists
// Update FieldVector with the new compressed buffers
fieldVector.clear(); // Clear existing buffers (to prevent memory leaks)
fieldVector.loadFieldBuffers(fieldNode,java.util.Arrays.asList(validityBuffer, offsetBuffer, compressedBuf)); // Load the new (compressed) buffers
fieldVector.setValueCount(fieldVectorValueCount); // Reset value count after buffer replacement
Field field1 = fieldVector.getField();
```
and i check the buffers again then buffers reflect old value.
please advise .
### Component(s)
Java
貢獻指南
研究方向
檢視涉及 getFieldBuffers()、clear() 和 loadFieldBuffers() 的 VectorSchemaRoot 與 FieldVector API,首先確認它們的緩衝區所有權與替換契約。重現所示序列,並驗證產生的向量公開的是 compressedBuf 還是原始緩衝區;完成的標準是找出一條受支援的更新路徑,或透過一個針對性測試確認此行為。
由索引模型根據 Issue 內容生成。
評估
- 技術堆疊
- java
- 領域
- data-engineering
- Issue 類型
- 缺陷
- 難度
- 4/5
- 預估耗時
- 3-5 天
- 活躍度
- 停滯
- 描述清晰度
- 需要釐清
- 新手友好度
- 25/100