Spark portable streaming: failing window-related tests
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
When running validatesPortableRunnerStreaming on Spark, the following tests fail. They all seem to be related to windowing. The underlying issue may have to do with incorrect handling of watermarks (they may not be getting held back properly).
Fails from timeout:
PAssertTest. testWindowedContainsInAnyOrder
PAssertTest. testWindowedSerializablePredicate
WindowTest. testNoWindowFnDoesNotReassignWindows
Fails from incorrect assertion:
CombineTest$BasicTests. testHotKeyCombining
Fails from empty assertion ("Expected: iterable with items [*] in any order
but: no item matches: * in []"):
CombineTest$WindowingTests. testFixedWindowsCombine
CombineTest$WindowingTests. testSessionsCombine
ReshuffleTest. testReshuffleAfterFixedWindowsAndGroupByKey
ReshuffleTest. testReshuffleAfterSessionsAndGroupByKey
ReshuffleTest. testReshuffleAfterSlidingWindowsAndGroupByKey
CoGroupByKeyTest. testCoGroupByKeyWithWindowing
WindowingTest. testMergingWindowing
WindowingTest. testNonPartitioningWindowing
WindowingTest. testPartitioningWindowing
WindowingTest. testWindowPreservation
Imported from Jira [BEAM-10784](https://issues.apache.org/jira/browse/BEAM-10784). Original Jira may contain additional context.
Reported by: annaqin.
Contributor guide
Research direction
Start by running validatesPortableRunnerStreaming on Spark and reviewing the listed failures in PAssertTest, WindowTest, CombineTest, ReshuffleTest, CoGroupByKeyTest, and WindowingTest. Investigate the reported windowing and watermark behavior, using the linked Jira issue for additional context. Done means the listed portable streaming tests pass on Spark.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- stream-processing, testing-qa
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100