apache / apache/hop

[Feature Request]: BigQuery input needs support for RECORD data type

Open
#5,064 0 comments 0 reactions 0 assignees View on GitHub
Beam P3
Dominant language
Java
Stars
1.5k
Forks
476
Avg merge
19h
Merged PRs (30d)
227

Description

### What would you like to happen?

When querying tabes in the Beam BigQuery Input transform, it's not possible to detect or query fields of type RECORD. It would be nice if Hop handled these nested fields with more grace. One possibility would be to give back the record fields as JSON strings.

### Issue Priority

Priority: 3

### Issue Component

Component: Beam

Contributor guide

Open the contributing guide

Research direction

Start at the Beam BigQuery Input transform and inspect how BigQuery schemas and fields are detected and queried. Determine how RECORD fields should be represented, then add coverage showing that nested fields can be queried and handled consistently, with JSON strings as one proposed output.

Written by the indexing model from the issue text.

Assessment

Tech stack
google-cloud, java
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.