apache / apache/beam

[SQL] Support schema qualifiers for field names

Open
#18,647 0 comments 0 reactions 0 assignees View on GitHub
bug dsl P3 sql
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
2d 2h
Merged PRs (30d)
205

Description

Currently there are utility methods in BeamRecord to get field values by name, e.g. BeamRecord.getFieldValue(String name). Internally they call fieldNamesArrayList.indexOf(fieldName) to find the index of the field name.

This works as long as there is only one field with such name in the record. But when joining 2 records you can end up with duplicate field names, and without any means of distinguishing them or getting a value from specific field by name. We don't keep any metadata in BeamRecordType to help identify a field in this case. 

It feels that this can lead to obscure bugs.

We probably should keep more detailed schema information attached to the fields, so that we could reference them using qualifiers like "[schemaA].[pcollectionB].[fieldC]".

 

Imported from Jira [BEAM-3574](https://issues.apache.org/jira/browse/BEAM-3574). Original Jira may contain additional context.
Reported by: kedin.

Contributor guide

Open the contributing guide

Research direction

Start by reading BeamRecord.getFieldValue(String name), the fieldNamesArrayList lookup, and BeamRecordType to understand how field metadata is currently represented. Check the SQL-related BeamRecord usage and tests for joined records. Done means duplicate field names can be distinguished and retrieved through the requested schema, PCollection, and field qualifiers.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, sql
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.