apache / apache/arrow-java

[Java] MapVector cannot be loaded via IPC

Đang mở
#71 8 bình luận 1 reaction 1 người được giao Được @vibhatha nhận Xem trên GitHub
Type: bug
Ngôn ngữ chính
Java
Star
94
Fork
152
Merge trung bình
3 ngày 16 giờ
Pull request đã merge (30 ngày)
11

Mô tả

### Describe the bug, including details regarding any error messages, version, and platform.

Referring to the stackoverflow filed issue: https://stackoverflow.com/questions/77878272/apache-arrow-not-all-nodes-and-buffers-were-consumed-error-when-writing-a-map

The following code would yield an error;

```java
File file = new File("test.arrow");

Field keyField = new Field("id", FieldType.notNullable(new ArrowType.Int(64, true)),
Collections.emptyList());
Field valueField = new Field("value", FieldType.nullable(new ArrowType.Int(64, true)), Collections.emptyList());
Field structField =
new Field("entry", FieldType.notNullable(ArrowType.Struct.INSTANCE), List.of(keyField, valueField));
Field mapIntToIntField = new Field("mapFieldIntToInt", FieldType.notNullable(new ArrowType.Map(false)), List.of(structField));

Schema schema = new Schema(Arrays.asList(mapIntToIntField));

System.out.println("Writing...");

try (BufferAllocator allocator = new RootAllocator()) {
try (
VectorSchemaRoot vectorSchemaRoot = VectorSchemaRoot.create(schema, allocator);
MapVector mapVector = (MapVector) vectorSchemaRoot.getVector("mapFieldIntToInt")) {
UnionMapWriter mapWriter = mapVector.getWriter();
mapWriter.setPosition(0);
mapWriter.startMap();
for (int i = 0; i < 3; i++) {
mapWriter.startEntry();
mapWriter.key().bigInt().writeBigInt(i);
mapWriter.value().bigInt().writeBigInt(i * 7);
mapWriter.endEntry();
}
mapWriter.endMap();
mapWriter.setValueCount(1);
vectorSchemaRoot.setRowCount(1);

System.out.println(vectorSchemaRoot.getFieldVectors().size());
System.out.println("vectorSchemaRoot.getVector(0): " + vectorSchemaRoot.getVector(0));

try (
FileOutputStream fileOutputStream = new FileOutputStream(file);
ArrowFileWriter writer = new ArrowFileWriter(vectorSchemaRoot, null, fileOutputStream.getChannel())) {
writer.start();
writer.writeBatch();
writer.end();
} catch (IOException e) {
e.printStackTrace();
}
}
}

System.out.println("Reading...");

try(
BufferAllocator rootAllocator = new RootAllocator();
FileInputStream fileInputStream = new FileInputStream(file);
ArrowFileReader reader = new ArrowFileReader(fileInputStream.getChannel(), rootAllocator)
){
System.out.println("Record batches in file: " + reader.getRecordBlocks().size());
for (ArrowBlock arrowBlock : reader.getRecordBlocks()) {
boolean loaded = reader.loadRecordBatch(arrowBlock);
System.out.println(loaded);
VectorSchemaRoot vectorSchemaRootRecover = reader.getVectorSchemaRoot();
System.out.print(vectorSchemaRootRecover.contentToTSVString());
}
} catch (IOException e) {
e.printStackTrace();
}
```

Error

```bash
Exception in thread "main" java.lang.IllegalArgumentException: not all nodes, buffers and variadicBufferCounts were consumed. nodes: [ArrowFieldNode [length=3, nullCount=0]] buffers: [ArrowBuf[24], address:123230812873128, capacity:1, ArrowBuf[25], address:123230812873136, capacity:24] variadicBufferCounts: []
at org.apache.arrow.vector.VectorLoader.load(VectorLoader.java:98)
at org.apache.arrow.vector.ipc.ArrowReader.loadRecordBatch(ArrowReader.java:214)
at org.apache.arrow.vector.ipc.ArrowFileReader.loadNextBatch(ArrowFileReader.java:166)
at org.apache.arrow.vector.ipc.ArrowFileReader.loadRecordBatch(ArrowFileReader.java:192)
```

### Component(s)

Java

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.