apache / apache/beam

Add the ability to override values set with @Default.YYY on PipelineOptions

Open
#18,347 0 comments 0 reactions 0 assignees View on GitHub
core improvement java P3
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

Users may want to set a value only if the value isn't the default.

When a user calls *PipelineOptionsFactory.fromArgs(args).as(Options.class)* they cannot tell if a user set the value from the command-line or whether the value is a default that was generated through the usage of the *@Default.YYY* annotations.

There is currently no method to introspect whether a default is set and we could:
* add a *isOptionSet()* method, but this adds many methods to the interfaces
* ask whether something is the default or set by name or by method reference, *options.isSet("options" / method)*, but is brittle to any name changes in options.
* allowing users to override with another interface defining a new *@Default.YYY* is difficult because:
** multiple inheritance makes choosing a default difficult:
```

MyPipelineOptions extends OptionsA, OptionsB { }
OptionsA {
@Default.String("A")
String getString();
}
OptionsB
{
@Default.String("B")
String getString();
}

```

** even if the was little ambiguity initially with MyPipelineOptions defined as:
```

MyPipelineOptions extends OptionsA, OptionsB {
@Default.String("C")
String getString();
}

```

defaults are resolved lazily on first access and it would be strange if the default resolved depending on the order of "as" calls.
```

options.as(MyPipelineOptions.class).getString()
options.as(OptionsA.class).getString()
options.as(OptionsB.class).getString()
would
all provide different answers.

```

Finally that leaves us with an option to add support for merging PipelineOptions (as long as they are compatible).
```

PipelineOptions PipelineOptionsFactory.merge(PipelineOptions ... options)

```

where all subsequent options overwrite any prior set options and the empty array returning the default *PipelineOptionsFactory.create()*

Imported from Jira [BEAM-2261](https://issues.apache.org/jira/browse/BEAM-2261). Original Jira may contain additional context.
Reported by: lcwik.

Contributor guide

Open the contributing guide

Research direction

Start by reading the existing PipelineOptionsFactory and PipelineOptions APIs, especially how defaults are resolved and options are converted with as(). Define the merge behavior for compatible options, including later values overriding earlier set values and an empty input producing PipelineOptionsFactory.create(); verify the behavior with relevant existing tests or new tests for these cases.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
api, backend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.