apache / apache/beam

Spark portable streaming: failing window-related tests

Open
#20,429 0 comments 0 reactions 0 assignees View on GitHub
bug P3 portability runners spark
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

When running validatesPortableRunnerStreaming on Spark, the following tests fail. They all seem to be related to windowing. The underlying issue may have to do with incorrect handling of watermarks (they may not be getting held back properly).

Fails from timeout:
PAssertTest. testWindowedContainsInAnyOrder
PAssertTest. testWindowedSerializablePredicate
WindowTest. testNoWindowFnDoesNotReassignWindows

Fails from incorrect assertion:
CombineTest$BasicTests. testHotKeyCombining

Fails from empty assertion ("Expected: iterable with items [*] in any order
but: no item matches: * in []"):
CombineTest$WindowingTests. testFixedWindowsCombine
CombineTest$WindowingTests. testSessionsCombine
ReshuffleTest. testReshuffleAfterFixedWindowsAndGroupByKey
ReshuffleTest. testReshuffleAfterSessionsAndGroupByKey
ReshuffleTest. testReshuffleAfterSlidingWindowsAndGroupByKey
CoGroupByKeyTest. testCoGroupByKeyWithWindowing
WindowingTest. testMergingWindowing
WindowingTest. testNonPartitioningWindowing
WindowingTest. testPartitioningWindowing
WindowingTest. testWindowPreservation

Imported from Jira [BEAM-10784](https://issues.apache.org/jira/browse/BEAM-10784). Original Jira may contain additional context.
Reported by: annaqin.

Contributor guide

Open the contributing guide

Research direction

Start by running validatesPortableRunnerStreaming on Spark and reviewing the listed failures in PAssertTest, WindowTest, CombineTest, ReshuffleTest, CoGroupByKeyTest, and WindowingTest. Investigate the reported windowing and watermark behavior, using the linked Jira issue for additional context. Done means the listed portable streaming tests pass on Spark.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
stream-processing, testing-qa
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.