apache / apache/beam

Steer people towards ParDo, SDF, instead of the original Source framework

Open
#20,729 0 comments 0 reactions 0 assignees View on GitHub
improvement P3 website
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

People still write sources, where 90% of the time they shouldn't. We tell them [not to](https://beam.apache.org/documentation/io/developing-io-overview/), but we should do so more effectively. In particular, the instructions for the ParDo alternative suffer from not being able to name Reshuffle explicitly, when it's exactly what should be used here. It should also mention that the ParDo needs to be seeded by a Create step or similar.

A big issue here is that Sources are called "Sources". When a new developer is looking to author a pipeline, this is the first place they will look, especially if they're just scanning or searching through documentation. We need to aggressively counteract the gravity of the current naming scheme.

Suggestion: Improve the documentation mentioned above, and update the Javadoc for BoundedSource, etc., to steer people away from it. If they are part of the small collection of power users who need a source, they'll be okay.

Suggestions for future work:
- Consider deprecating source framework in favor of SDF.
- Point to SDF docs (and simplify SDF docs)
- Also many users can simply just use FileIO.matchAll followed by a ParDo. Recommend those types of alternatives.

Assigning this to [~boyuanz] anyone could help here.
/cc [~kenn] [~chamikara] [~reuvenlax] [~robertwb] [~rtnguyen] [~dcavazos]

Imported from Jira [BEAM-11633](https://issues.apache.org/jira/browse/BEAM-11633). Original Jira may contain additional context.
Reported by: altay.

Contributor guide

Open the contributing guide

Research direction

Read the linked developing-io-overview documentation first, then inspect the BoundedSource and related source-framework Javadocs. Done means the documentation and API guidance consistently direct typical users to ParDo, SDF, or FileIO alternatives, including the relevant Reshuffle and Create guidance, while still explaining when a source is appropriate.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
documentation
Issue type
Documentation
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.