apache / apache/hop

[Feature Request]: Support for Objects and Array data types

Open
#2,404 2 comments 0 reactions 0 assignees View on GitHub
awaiting triage Metadata P3
Dominant language
Java
Stars
1.5k
Forks
476
Avg merge
19h
Merged PRs (30d)
227

Description

### What would you like to happen?

Migration: https://issues.apache.org/jira/browse/HOP-2476

As a developer I come across objects and arrays stored as part of a record fairly regularly. I'd like Hop to natively support them with dedicated data types so that extracting values becomes easier (no need to use any scripting or coding).

In example, similarly to Trifacta, Hop could offer easy to use functions like these ones then:

1) "Flatten array values into new rows":
[\{"type":"primary","colour":"green","product_number":3899},\{"type":"secondary","colour":"red","product_number":3899}]

returns two rows:

{"type":"primary","colour":"green","product_number":3899} {"type":"secondary","colour":"red","product_number":3899}

2) "Unnest values into new columns". The unnest transformation takes a nested key-value pair in the object and extracts it to a new column. e.g. unnest product_number:{"type":"secondary","colour":"red","product_number":3899} => 3899

### Issue Priority

Priority: 3

### Issue Component

Component: Metadata

Contributor guide

Open the contributing guide

Research direction

No source files, tests, or entry points are named. Start by reviewing the Metadata component and migration HOP-2476, then clarify the design for native object and array types and their flattening and unnesting operations; done would require an agreed implementation scope and validation approach.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.