[Java] VectorSchemaRoot convertion into ArrowArrayStream
- Dominant language
- Java
- Stars
- 94
- Forks
- 152
- Avg merge
- 3d 16h
- Merged PRs (30d)
- 11
Description
### Describe the enhancement requested
DuckDB support ArrowArrayStream as virtual table. JDBC driver of DuckDB has "register" ArrowStream method on DuckDBConnection.
VectorSchemaRoot convertion into ArrowArrayStream object will allows perform complex SQL transformation with CTE,window function on existing Arrow data from java process without data format transformation and also as workaround to write parquet file from VectorSchemaRoot.
```
try (DuckDBConnection connection = (DuckDBConnection) DriverManager.getConnection("jdbc:duckdb:")){
connection.registerArrowStream("arrow_stream", arrowArrayStream);
try (Statement preparedStatement = connection.createStatement()){
preparedStatement.executeUpdate("COPY (SELECT * FROM arrow_stream) TO '"+filePath+"' (FORMAT 'PARQUET', CODEC 'ZSTD')");
}
}
```
@lidavidm do you have idea how to implement such conversion?
### Component(s)
Java
Contributor guide
Research direction
Start with the Java component and the issue's DuckDB JDBC example, then inspect the VectorSchemaRoot and ArrowArrayStream APIs together with DuckDBConnection.registerArrowStream. Done means a VectorSchemaRoot can be converted into an ArrowArrayStream that the shown registration and SQL workflow can consume without format transformation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100