apache / apache/beam

Create a generator of finite-but-unbounded PCollection's for integration testing

Open
#18,686 0 comments 0 reactions 0 assignees View on GitHub
core java new feature P3
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
2d 2h
Merged PRs (30d)
205

Description

Several IOs have features that exhibit nontrivial behavior when writing unbounded PCollection's - e.g. WriteFiles with windowed writes; BigQueryIO. We need to be able to write integration tests for these features.

Currently we have two ways to generate an unbounded PCollection without reading from a real-world external streaming system such as pubsub or kafka:

1) TestStream, which only works in direct runner - sufficient for some tests but not all: definitely not sufficient for large-scale tests or for tests that need to interact with a real instance of the external system (e.g. BigQueryIO). It is also quite verbose to use.
2) GenerateSequence.from(0) without a .to(), which returns an infinite amount of data.

GenerateSequence.from(a).to(b) returns a finite amount of data, but returns it as a bounded PCollection, and doesn't report the watermark.

I think the right thing to do here, for now, is to make GenerateSequence.from(a).to(b) have an option (e.g. ".asUnbounded()", where it will return an unbounded PCollection, go through UnboundedSource (or potentially via SDF in runners that support it), and track the watermark properly (or via a configurable watermark fn).

Imported from Jira [BEAM-3323](https://issues.apache.org/jira/browse/BEAM-3323). Original Jira may contain additional context.
Reported by: jkff.

Contributor guide

Open the contributing guide

Research direction

Start at the GenerateSequence.from(a).to(b) entry point and compare its bounded behavior with TestStream and the unbounded GenerateSequence form. Define what is needed for WriteFiles and BigQueryIO integration tests, including unbounded output and watermark tracking, then validate the behavior with relevant integration tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
stream-processing, testing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.