apache / apache/arrow-java

[Java] Map cannot be written to a Parquet file using DatasetFileWriter

Offen
#160 9 Kommentare 0 Reaktionen 1 zugewiesene Person Beansprucht von @vibhatha Auf GitHub ansehen
Vorherrschende Sprache
Java
Sterne
94
Forks
152
Ø Merge
3 T. 16 Std.
Gemergte PRs (30 T.)
11

Beschreibung

### Describe the bug, including details regarding any error messages, version, and platform.

Trying to write a map column to a parquet file using DatasetFileWriter results in the following error:
`Runtime java.lang.IllegalArgumentException: not all nodes and buffers were consumed. nodes: [ArrowFieldNode [length=2, nullCount=1], ArrowFieldNode [length=0, nullCount=0], ArrowFieldNode [length=0, nullCount=0]] buffers: [ArrowBuf[509], address:140461672595576, capacity:0, ArrowBuf[510], address:140461672595576, capacity:1, ArrowBuf[511], address:140461672595584, capacity:8, ArrowBuf[512], address:140461672595592, capacity:0, ArrowBuf[513], address:140461672595592, capacity:0, ArrowBuf[514], address:140461672595592, capacity:0]
at org.apache.arrow.vector.VectorLoader.load(VectorLoader.java:89)
at org.apache.arrow.vector.ipc.ArrowReader.loadRecordBatch(ArrowReader.java:220)
at org.apache.arrow.vector.ipc.ArrowStreamReader.loadNextBatch(ArrowStreamReader.java:161)
at org.apache.arrow.c.ArrayStreamExporter$ExportedArrayStreamPrivateData.getNext(ArrayStreamExporter.java:72)`

This happens when the data-to-be-written comes from an ArrowFileReader or ArrowStreamReader. This problem may affect more file formats than parquet (and might actually be related to the DatasetFileWriter in general).

### Component(s)

Java, Parquet

Beitragsleitfaden

Beitragsleitfaden öffnen

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.