apache / apache/arrow

Arrow Flight Stream Writing vector Schema Root from Batches

Open
#14,812 10 comments 0 reactions 0 assignees View on GitHub
Component: FlightRPC Type: usage
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 18h
Merged PRs (30d)
91

Description

### Describe the usage question you have. Please include as many useful details as possible.

I am trying to write the Iceberg data to Arrow flight server using the example described here https://arrow.apache.org/cookbook/java/flight.html I am reading files from Icebereg

`VectorSchemaRoot vectorSchemaRoot = VectorSchemaRoot.create(tempSchemaRoot.getSchema(), allocator);
listener = flightClient.startPut(
FlightDescriptor.path("profiles"),
vectorSchemaRoot, new AsyncPutListener());
try (VectorizedTableScanIterable itr = new VectorizedTableScanIterable(filteredScan, 10000, false); ) {
vectorSchemaRoot.allocateNew();
for (ColumnarBatch batch : itr) {
VectorSchemaRoot root = batch.createVectorSchemaRootFromVectors();
List fieldVectors = root.getFieldVectors();
vectorSchemaRoot = VectorSchemaRoot.of(fieldVectors.toArray(new FieldVector[0]));
listener.putNext();
break;
}
}catch(Exception e){
System.out.println(e);
}
listener.completed();
listener.getResult();`
@lwhite1
When try to read the data back I don't get the records. Is there any example on how to create a streaming write to Arrow Flight from multiple batches of vectorschema root that you get from reading a file.

### Component(s)

FlightRPC

Contributor guide

Open the contributing guide

Research direction

Start with the Java Flight cookbook example linked in the issue, then compare its FlightClient.startPut and VectorSchemaRoot usage with the shown VectorizedTableScanIterable and ColumnarBatch flow. Clarify how multiple batches should be streamed and what a reader should observe when the write is complete; the documentation is done when it provides a working multi-batch example.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
api
Issue type
Documentation
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.