apache / apache/beam

Source.getDefaultOutputCoder() - add a version that allows access to a CoderRegistry

Open
#18,177 0 comments 0 reactions 0 assignees View on GitHub
bug core java P3
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

The Source API's getDefaultOutputCoder currently takes no parameters. This means that you can't access the coder registry to ask for coders based on the type.

The PTransform API's version of getDefaultOutputCoder has 3 different form, which do allow access to the coder registry if necessary.

However, while working on BEAM-1040 (Hadoop IO connector), we realized that a source does not always know it's output type, and thus will need to calculate the coder at run time. To do this, it would be helpful to have a reference to the CoderRegistry - we should look at what we're doing on PTransform and decide if we should port that over onto Source.

Imported from Jira [BEAM-1171](https://issues.apache.org/jira/browse/BEAM-1171). Original Jira may contain additional context.
Reported by: sisk.

Contributor guide

Open the contributing guide

Research direction

Start by comparing the Source API's getDefaultOutputCoder with the three PTransform variants, focusing on how each exposes CoderRegistry. Review the BEAM-1040 Hadoop IO connector context and the original BEAM-1171 discussion. Done means deciding the appropriate Source API shape and documenting or implementing registry access for sources whose output type is determined at runtime.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
api, backend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.