apache / apache/beam

Support Custom Windows in Spark runner

Open
#18,501 0 comments 0 reactions 0 assignees View on GitHub
bug P3 runners spark
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
2d 5h
Merged PRs (30d)
204

Description

If we extend `IntervalWindow` and we try to merge these custom windows like in this PR:
https://github.com/apache/beam/pull/3286
Then spark runner fails with

`org.apache.beam.sdk.Pipeline$PipelineExecutionException: java.lang.ClassCastException: org.apache.beam.sdk.transforms.windowing.IntervalWindow cannot be cast to org.apache.beam.sdk.transforms.windowing.MergingCustomWindowsTest$CustomWindow`

It seems to be because of the cast to `IntervalWindow` there: https://github.com/apache/beam/blob/master/runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkGlobalCombineFn.java#L111

Imported from Jira [BEAM-2499](https://issues.apache.org/jira/browse/BEAM-2499). Original Jira may contain additional context.
Reported by: echauchot.

Contributor guide

Open the contributing guide

Research direction

Start in runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkGlobalCombineFn.java at line 111, then review the custom-window merge scenario described in the issue and linked PR. Reproduce the Spark runner failure with an IntervalWindow subclass and verify that custom windows merge without the reported ClassCastException.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
distributed-systems
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.