apache / apache/arrow-java

[Java] MapVector cannot be loaded via IPC

未關閉
#71 8 則留言 1 個 reaction 已指派 1 人 已被 @vibhatha 認領 在 GitHub 檢視
Type: bug
主要語言
Java
星號
94
分支
152
平均合併
3 天 16 小時
30 天內合併 PR
11

描述

### Describe the bug, including details regarding any error messages, version, and platform.

Referring to the stackoverflow filed issue: https://stackoverflow.com/questions/77878272/apache-arrow-not-all-nodes-and-buffers-were-consumed-error-when-writing-a-map

The following code would yield an error;

```java
File file = new File("test.arrow");

Field keyField = new Field("id", FieldType.notNullable(new ArrowType.Int(64, true)),
Collections.emptyList());
Field valueField = new Field("value", FieldType.nullable(new ArrowType.Int(64, true)), Collections.emptyList());
Field structField =
new Field("entry", FieldType.notNullable(ArrowType.Struct.INSTANCE), List.of(keyField, valueField));
Field mapIntToIntField = new Field("mapFieldIntToInt", FieldType.notNullable(new ArrowType.Map(false)), List.of(structField));

Schema schema = new Schema(Arrays.asList(mapIntToIntField));

System.out.println("Writing...");

try (BufferAllocator allocator = new RootAllocator()) {
try (
VectorSchemaRoot vectorSchemaRoot = VectorSchemaRoot.create(schema, allocator);
MapVector mapVector = (MapVector) vectorSchemaRoot.getVector("mapFieldIntToInt")) {
UnionMapWriter mapWriter = mapVector.getWriter();
mapWriter.setPosition(0);
mapWriter.startMap();
for (int i = 0; i < 3; i++) {
mapWriter.startEntry();
mapWriter.key().bigInt().writeBigInt(i);
mapWriter.value().bigInt().writeBigInt(i * 7);
mapWriter.endEntry();
}
mapWriter.endMap();
mapWriter.setValueCount(1);
vectorSchemaRoot.setRowCount(1);

System.out.println(vectorSchemaRoot.getFieldVectors().size());
System.out.println("vectorSchemaRoot.getVector(0): " + vectorSchemaRoot.getVector(0));

try (
FileOutputStream fileOutputStream = new FileOutputStream(file);
ArrowFileWriter writer = new ArrowFileWriter(vectorSchemaRoot, null, fileOutputStream.getChannel())) {
writer.start();
writer.writeBatch();
writer.end();
} catch (IOException e) {
e.printStackTrace();
}
}
}

System.out.println("Reading...");

try(
BufferAllocator rootAllocator = new RootAllocator();
FileInputStream fileInputStream = new FileInputStream(file);
ArrowFileReader reader = new ArrowFileReader(fileInputStream.getChannel(), rootAllocator)
){
System.out.println("Record batches in file: " + reader.getRecordBlocks().size());
for (ArrowBlock arrowBlock : reader.getRecordBlocks()) {
boolean loaded = reader.loadRecordBatch(arrowBlock);
System.out.println(loaded);
VectorSchemaRoot vectorSchemaRootRecover = reader.getVectorSchemaRoot();
System.out.print(vectorSchemaRootRecover.contentToTSVString());
}
} catch (IOException e) {
e.printStackTrace();
}
```

Error

```bash
Exception in thread "main" java.lang.IllegalArgumentException: not all nodes, buffers and variadicBufferCounts were consumed. nodes: [ArrowFieldNode [length=3, nullCount=0]] buffers: [ArrowBuf[24], address:123230812873128, capacity:1, ArrowBuf[25], address:123230812873136, capacity:24] variadicBufferCounts: []
at org.apache.arrow.vector.VectorLoader.load(VectorLoader.java:98)
at org.apache.arrow.vector.ipc.ArrowReader.loadRecordBatch(ArrowReader.java:214)
at org.apache.arrow.vector.ipc.ArrowFileReader.loadNextBatch(ArrowFileReader.java:166)
at org.apache.arrow.vector.ipc.ArrowFileReader.loadRecordBatch(ArrowFileReader.java:192)
```

### Component(s)

Java

貢獻指南

開啟貢獻指南

評估

這個 Issue 還沒有評估資料。

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。