TestPipeline / PipelineOptionsFactory crashes on to unregistered options
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 205
Description
When `TestSparkPipelineOptions` is not registered, this failure occurs on all tests using `TestPipeline`:
```
java.lang.IllegalArgumentException: Class interface org.apache.beam.sdk.testing.TestPipelineOptions
missing a property named 'forceStreaming'.
at org.apache.beam.sdk.options.PipelineOptionsFactory.parseObjects(PipelineOptionsFactory.java:1573)
at
org.apache.beam.sdk.options.PipelineOptionsFactory.access$400(PipelineOptionsFactory.java:104)
at org.apache.beam.sdk.options.PipelineOptionsFactory$Builder.as(PipelineOptionsFactory.java:289)
at
org.apache.beam.sdk.testing.TestPipeline.testingPipelineOptions(TestPipeline.java:403)
at org.apache.beam.sdk.testing.TestPipeline.create(TestPipeline.java:264)
at
org.apache.beam.runners.spark.translation.streaming.StreamingSourceMetricsTest.(StreamingSourceMetricsTest.java:49)
```
However, it is (I believe) our recommended best practice to not register such options as they are not intended for users. They should just be "cast" via `as` by the runner that wants to use them. Further, `TestSparkPipelineOptions` cannot be registered due to pulling in a hamcrest dependency (see BEAM-2237).
The user doesn't specify this option, and the runner doesn't mutate it. This should probably not crash.
Imported from Jira [BEAM-2238](https://issues.apache.org/jira/browse/BEAM-2238). Original Jira may contain additional context.
Reported by: kenn.
Contributor guide
Research direction
Start with PipelineOptionsFactory.java at parseObjects and TestPipeline.java at testingPipelineOptions/create, using the reported StreamingSourceMetricsTest.java stack trace to reproduce the failure. Compare the handling of unregistered TestPipelineOptions and TestSparkPipelineOptions; done means tests using TestPipeline no longer fail when the option is absent and unregistered.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100