[SQL] Support schema qualifiers for field names
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 205
Description
Currently there are utility methods in BeamRecord to get field values by name, e.g. BeamRecord.getFieldValue(String name). Internally they call fieldNamesArrayList.indexOf(fieldName) to find the index of the field name.
This works as long as there is only one field with such name in the record. But when joining 2 records you can end up with duplicate field names, and without any means of distinguishing them or getting a value from specific field by name. We don't keep any metadata in BeamRecordType to help identify a field in this case.
It feels that this can lead to obscure bugs.
We probably should keep more detailed schema information attached to the fields, so that we could reference them using qualifiers like "[schemaA].[pcollectionB].[fieldC]".
Imported from Jira [BEAM-3574](https://issues.apache.org/jira/browse/BEAM-3574). Original Jira may contain additional context.
Reported by: kedin.
Contributor guide
Research direction
Start by reading BeamRecord.getFieldValue(String name), the fieldNamesArrayList lookup, and BeamRecordType to understand how field metadata is currently represented. Check the SQL-related BeamRecord usage and tests for joined records. Done means duplicate field names can be distinguished and retrieved through the requested schema, PCollection, and field qualifiers.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, sql
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100