Running tests of dynamic work rebalancing
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
SourceTestUtils has an exhaustive test for splitAtFraction. However, we should also add a runner-independent test that, for example, can verify that dynamic work rebalancing is actually invoked for a new BoundedSource.
An example of this test was proposed in https://github.com/GoogleCloudPlatform/DataflowJavaSDK/blob/master/sdk/src/test/java/com/google/cloud/dataflow/sdk/transforms/GroupByKeyTest.java#L435 , but was specific to the Google Cloud Dataflow service.
Imported from Jira [BEAM-75](https://issues.apache.org/jira/browse/BEAM-75). Original Jira may contain additional context.
Reported by: dhalperi.
Contributor guide
Research direction
Start by reading the exhaustive splitAtFraction test in SourceTestUtils and compare it with the runner-specific example in GroupByKeyTest. The work is done when a runner-independent test verifies that dynamic work rebalancing is invoked for a new BoundedSource.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, testing
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100