Support for testing how soon output is emitted
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 205
Description
I ran into this issue when testing Splittable DoFn. My intention is, it should behave exactly like a DoFn - i.e. emit output immediately when it receives input, regardless of the windowing/triggering strategy of the input (even though SDF has a GBK internally).
However, currently the SDK doesn't have facilities for testing that. TestStream allows controlling the timing of the input, but there's nothing to capture timing of the output. Moreover, timing of the output is unspecified by the model because triggers technically only enable firing, but do not force it (i.e. they are a lower bound on when output will be emitted).
I'm not sure what's the best way to address this. E.g., perhaps, PaneInfo could include a field "since when was this pane enabled to fire" (regardless of when it really fired)?
Imported from Jira [BEAM-988](https://issues.apache.org/jira/browse/BEAM-988). Original Jira may contain additional context.
Reported by: jkff.
Contributor guide
Research direction
Start by reading the TestStream, PaneInfo, and Splittable DoFn testing entry points mentioned in the issue, then review how windowing and triggers affect observable output. The work is done when the SDK provides a decided, documented way to test when output becomes eligible and a test demonstrates the intended behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- stream-processing, testing-qa
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100