Arrow Flight Stream Writing vector Schema Root from Batches
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 91
Description
### Describe the usage question you have. Please include as many useful details as possible.
I am trying to write the Iceberg data to Arrow flight server using the example described here https://arrow.apache.org/cookbook/java/flight.html I am reading files from Icebereg
`VectorSchemaRoot vectorSchemaRoot = VectorSchemaRoot.create(tempSchemaRoot.getSchema(), allocator);
listener = flightClient.startPut(
FlightDescriptor.path("profiles"),
vectorSchemaRoot, new AsyncPutListener());
try (VectorizedTableScanIterable itr = new VectorizedTableScanIterable(filteredScan, 10000, false); ) {
vectorSchemaRoot.allocateNew();
for (ColumnarBatch batch : itr) {
VectorSchemaRoot root = batch.createVectorSchemaRootFromVectors();
List fieldVectors = root.getFieldVectors();
vectorSchemaRoot = VectorSchemaRoot.of(fieldVectors.toArray(new FieldVector[0]));
listener.putNext();
break;
}
}catch(Exception e){
System.out.println(e);
}
listener.completed();
listener.getResult();`
@lwhite1
When try to read the data back I don't get the records. Is there any example on how to create a streaming write to Arrow Flight from multiple batches of vectorschema root that you get from reading a file.
### Component(s)
FlightRPC
Contributor guide
Research direction
Start with the Java Flight cookbook example linked in the issue, then compare its FlightClient.startPut and VectorSchemaRoot usage with the shown VectorizedTableScanIterable and ColumnarBatch flow. Clarify how multiple batches should be streamed and what a reader should observe when the write is complete; the documentation is done when it provides a working multi-batch example.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- api
- Issue type
- Documentation
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100