apache / apache/hop

[Feature Request]: Create new Data Type Arrow Vectors

Open
#2,421 0 comments 0 reactions 0 assignees View on GitHub
awaiting triage Other P3
Dominant language
Java
Stars
1.5k
Forks
476
Avg merge
18h 32m
Merged PRs (30d)
216

Description

### What would you like to happen?

Migration: https://issues.apache.org/jira/browse/HOP-4121

I'd like to add a new core data type: an array of Apache Arrow FieldVectors. This allows for integrating with files and services that utilize Arrow-based data formats. Some examples include the Feather file format, Google BigQuery Storage Read API, and Neo4j's new streaming support with its Graph Data Science library (which leverages Apache Arrow Flight).

My initial proposal is to:

add the core ValueMeta type ValueMetaArrowVectors
add an ArrowEncode transformer for converting Hop row-wise data into batches of Arrow vectors
add an ArrowDecode transformer for converting Arrow vectors to one or many Hop rows

This will set a general purpose foundation for others to build plugins that can leverage Arrow, similar to how Avro support was recently merged.

I have not opened a PR yet as I'm still adding support for vectors of lists/arrays. (I've currently implemented support for vectors of scalars of Hop Integers, Numbers, Strings, Booleans, Dates, and Timestamps.) For those curious, my wip fork is available in my "arrow" branch at https://github.com/neo4j-field/hop

### Issue Priority

Priority: 3

### Issue Component

Component: Other

Contributor guide

Open the contributing guide

Research direction

The issue names no files or tests. Start by reviewing the proposed ValueMetaArrowVectors, ArrowEncode, and ArrowDecode entry points and the existing Avro support; use the linked arrow branch as context. Done means the core type and both conversions support the stated scalar and list/array cases and provide a foundation for Arrow-based integrations.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.