apache / apache/beam

TestPipeline / PipelineOptionsFactory crashes on to unregistered options

Open
#18,342 0 comments 0 reactions 0 assignees View on GitHub
bug core java P3
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
2d 2h
Merged PRs (30d)
205

Description

When `TestSparkPipelineOptions` is not registered, this failure occurs on all tests using `TestPipeline`:

```

java.lang.IllegalArgumentException: Class interface org.apache.beam.sdk.testing.TestPipelineOptions
missing a property named 'forceStreaming'.
at org.apache.beam.sdk.options.PipelineOptionsFactory.parseObjects(PipelineOptionsFactory.java:1573)
at
org.apache.beam.sdk.options.PipelineOptionsFactory.access$400(PipelineOptionsFactory.java:104)
at org.apache.beam.sdk.options.PipelineOptionsFactory$Builder.as(PipelineOptionsFactory.java:289)
at
org.apache.beam.sdk.testing.TestPipeline.testingPipelineOptions(TestPipeline.java:403)
at org.apache.beam.sdk.testing.TestPipeline.create(TestPipeline.java:264)
at
org.apache.beam.runners.spark.translation.streaming.StreamingSourceMetricsTest.(StreamingSourceMetricsTest.java:49)

```

However, it is (I believe) our recommended best practice to not register such options as they are not intended for users. They should just be "cast" via `as` by the runner that wants to use them. Further, `TestSparkPipelineOptions` cannot be registered due to pulling in a hamcrest dependency (see BEAM-2237).

The user doesn't specify this option, and the runner doesn't mutate it. This should probably not crash.

Imported from Jira [BEAM-2238](https://issues.apache.org/jira/browse/BEAM-2238). Original Jira may contain additional context.
Reported by: kenn.

Contributor guide

Open the contributing guide

Research direction

Start with PipelineOptionsFactory.java at parseObjects and TestPipeline.java at testingPipelineOptions/create, using the reported StreamingSourceMetricsTest.java stack trace to reproduce the failure. Compare the handling of unregistered TestPipelineOptions and TestSparkPipelineOptions; done means tests using TestPipeline no longer fail when the option is absent and unregistered.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.