apache / apache/beam

Implementation encoding position sdk-go

Open
#21,206 0 comments 0 reactions 0 assignees View on GitHub
go improvement P3
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

*Status*

The schema proto has an [encoding_position field](https://github.com/apache/beam/blob/2c619c81082839e054f16efee9311b9f74b6e436/model/pipeline/src/main/proto/schema.proto#L55) that is currently  implemented only on java and python sdk.

The intention of this field is that it indicates an alternative order for the fields to be encoded in by [beam:coder:row:v1 implementations](https://github.com/apache/beam/blob/1e60f383fb39b9ff8d44edcbe5357da4c1e52378/model/pipeline/src/main/proto/beam_runner_api.proto#L937-L990). The implementation encodes the fields in the order that they appear in the schema ignoring this field.

Imported from Jira [BEAM-13043](https://issues.apache.org/jira/browse/BEAM-13043). Original Jira may contain additional context.
Reported by: AlexRodriguez.

Contributor guide

Open the contributing guide

Research direction

Start with the encoding_position definition in model/pipeline/src/main/proto/schema.proto and the beam:coder:row:v1 description in model/pipeline/src/main/proto/beam_runner_api.proto. Trace the sdk-go row encoding entry point and compare its field ordering with the Java and Python SDK implementations. Done means Go row encoding honors encoding_position rather than only schema order, with corresponding coverage where the existing SDK tests reside.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.