Test Spark streaming pipelines with maxRecordsPerBatch
Open
good first issue
P3
runners
spark
tests
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
Add tests for Spark streaming pipelines with maxRecordsPerBatch configured in `SparkPipelineOptions`.
Imported from Jira [BEAM-2127](https://issues.apache.org/jira/browse/BEAM-2127). Original Jira may contain additional context.
Reported by: aviemzur.
Contributor guide
Research direction
Start by locating the Spark streaming pipeline tests and the SparkPipelineOptions definition, then inspect how maxRecordsPerBatch is currently configured. Add coverage for pipelines using that option and verify the tests demonstrate the expected batch behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- data-engineering, stream-processing, testing-qa
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100