Support Custom Windows in Spark runner
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 2d 5h
- Merged PRs (30d)
- 204
Description
If we extend `IntervalWindow` and we try to merge these custom windows like in this PR:
https://github.com/apache/beam/pull/3286
Then spark runner fails with
`org.apache.beam.sdk.Pipeline$PipelineExecutionException: java.lang.ClassCastException: org.apache.beam.sdk.transforms.windowing.IntervalWindow cannot be cast to org.apache.beam.sdk.transforms.windowing.MergingCustomWindowsTest$CustomWindow`
It seems to be because of the cast to `IntervalWindow` there: https://github.com/apache/beam/blob/master/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkGlobalCombineFn.java#L111
Imported from Jira [BEAM-2499](https://issues.apache.org/jira/browse/BEAM-2499). Original Jira may contain additional context.
Reported by: echauchot.
Contributor guide
Research direction
Start in runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkGlobalCombineFn.java at line 111, then review the custom-window merge scenario described in the issue and linked PR. Reproduce the Spark runner failure with an IntervalWindow subclass and verify that custom windows merge without the reported ClassCastException.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100