apache / apache/arrow-java

[Java] MapVector cannot be loaded via IPC

オープン
#71 コメント 8 件 リアクション 1 件 担当者 1 名 @vibhatha が担当を希望しています GitHub で見る
Type: bug
主要言語
Java
スター
94
フォーク
152
平均マージ
3日 16時間
マージ済み PR(30日)
11

説明

### Describe the bug, including details regarding any error messages, version, and platform.

Referring to the stackoverflow filed issue: https://stackoverflow.com/questions/77878272/apache-arrow-not-all-nodes-and-buffers-were-consumed-error-when-writing-a-map

The following code would yield an error;

```java
File file = new File("test.arrow");

Field keyField = new Field("id", FieldType.notNullable(new ArrowType.Int(64, true)),
Collections.emptyList());
Field valueField = new Field("value", FieldType.nullable(new ArrowType.Int(64, true)), Collections.emptyList());
Field structField =
new Field("entry", FieldType.notNullable(ArrowType.Struct.INSTANCE), List.of(keyField, valueField));
Field mapIntToIntField = new Field("mapFieldIntToInt", FieldType.notNullable(new ArrowType.Map(false)), List.of(structField));

Schema schema = new Schema(Arrays.asList(mapIntToIntField));

System.out.println("Writing...");

try (BufferAllocator allocator = new RootAllocator()) {
try (
VectorSchemaRoot vectorSchemaRoot = VectorSchemaRoot.create(schema, allocator);
MapVector mapVector = (MapVector) vectorSchemaRoot.getVector("mapFieldIntToInt")) {
UnionMapWriter mapWriter = mapVector.getWriter();
mapWriter.setPosition(0);
mapWriter.startMap();
for (int i = 0; i < 3; i++) {
mapWriter.startEntry();
mapWriter.key().bigInt().writeBigInt(i);
mapWriter.value().bigInt().writeBigInt(i * 7);
mapWriter.endEntry();
}
mapWriter.endMap();
mapWriter.setValueCount(1);
vectorSchemaRoot.setRowCount(1);

System.out.println(vectorSchemaRoot.getFieldVectors().size());
System.out.println("vectorSchemaRoot.getVector(0): " + vectorSchemaRoot.getVector(0));

try (
FileOutputStream fileOutputStream = new FileOutputStream(file);
ArrowFileWriter writer = new ArrowFileWriter(vectorSchemaRoot, null, fileOutputStream.getChannel())) {
writer.start();
writer.writeBatch();
writer.end();
} catch (IOException e) {
e.printStackTrace();
}
}
}

System.out.println("Reading...");

try(
BufferAllocator rootAllocator = new RootAllocator();
FileInputStream fileInputStream = new FileInputStream(file);
ArrowFileReader reader = new ArrowFileReader(fileInputStream.getChannel(), rootAllocator)
){
System.out.println("Record batches in file: " + reader.getRecordBlocks().size());
for (ArrowBlock arrowBlock : reader.getRecordBlocks()) {
boolean loaded = reader.loadRecordBatch(arrowBlock);
System.out.println(loaded);
VectorSchemaRoot vectorSchemaRootRecover = reader.getVectorSchemaRoot();
System.out.print(vectorSchemaRootRecover.contentToTSVString());
}
} catch (IOException e) {
e.printStackTrace();
}
```

Error

```bash
Exception in thread "main" java.lang.IllegalArgumentException: not all nodes, buffers and variadicBufferCounts were consumed. nodes: [ArrowFieldNode [length=3, nullCount=0]] buffers: [ArrowBuf[24], address:123230812873128, capacity:1, ArrowBuf[25], address:123230812873136, capacity:24] variadicBufferCounts: []
at org.apache.arrow.vector.VectorLoader.load(VectorLoader.java:98)
at org.apache.arrow.vector.ipc.ArrowReader.loadRecordBatch(ArrowReader.java:214)
at org.apache.arrow.vector.ipc.ArrowFileReader.loadNextBatch(ArrowFileReader.java:166)
at org.apache.arrow.vector.ipc.ArrowFileReader.loadRecordBatch(ArrowFileReader.java:192)
```

### Component(s)

Java

コントリビューションガイド

コントリビューションガイドを開く

評価

この issue はまだ評価されていません。

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。