SparkRunner should deduplicate when an UnboundedSource requires Deduping
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
The implementation of an Unbounded Read does not inspect the requiresDeduping property of the source, and as such does not appropriately deduplicate sources that require it.
https://github.com/apache/beam/blob/master/runners/spark/src/main/java/org/apache/beam/runners/spark/io/SparkUnboundedSource.java
Imported from Jira [BEAM-1725](https://issues.apache.org/jira/browse/BEAM-1725). Original Jira may contain additional context.
Reported by: tgroh.
Contributor guide
Research direction
Start in runners/spark/src/main/java/org/apache/beam/runners/spark/io/SparkUnboundedSource.java and trace how an unbounded read handles a source's requiresDeduping property. Confirm the expected behavior for sources requiring deduplication and verify that the Spark runner deduplicates them without changing sources that do not require it.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- stream-processing
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 45/100