[Feature Request]: Create new Data Type Arrow Vectors
- Dominant language
- Java
- Stars
- 1.5k
- Forks
- 476
- Avg merge
- 18h 32m
- Merged PRs (30d)
- 216
Description
### What would you like to happen?
Migration: https://issues.apache.org/jira/browse/HOP-4121
I'd like to add a new core data type: an array of Apache Arrow FieldVectors. This allows for integrating with files and services that utilize Arrow-based data formats. Some examples include the Feather file format, Google BigQuery Storage Read API, and Neo4j's new streaming support with its Graph Data Science library (which leverages Apache Arrow Flight).
My initial proposal is to:
add the core ValueMeta type ValueMetaArrowVectors
add an ArrowEncode transformer for converting Hop row-wise data into batches of Arrow vectors
add an ArrowDecode transformer for converting Arrow vectors to one or many Hop rows
This will set a general purpose foundation for others to build plugins that can leverage Arrow, similar to how Avro support was recently merged.
I have not opened a PR yet as I'm still adding support for vectors of lists/arrays. (I've currently implemented support for vectors of scalars of Hop Integers, Numbers, Strings, Booleans, Dates, and Timestamps.) For those curious, my wip fork is available in my "arrow" branch at https://github.com/neo4j-field/hop
### Issue Priority
Priority: 3
### Issue Component
Component: Other
Contributor guide
Research direction
The issue names no files or tests. Start by reviewing the proposed ValueMetaArrowVectors, ArrowEncode, and ArrowDecode entry points and the existing Avro support; use the linked arrow branch as context. Done means the core type and both conversions support the stated scalar and list/array cases and provide a foundation for Arrow-based integrations.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100