apache / apache/beam

Need ValueProvider that executes exactly once at pipeline runtime.

Open
#18,765 0 comments 0 reactions 0 assignees View on GitHub
bug dataflow P3 runners
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

When a dataflow is started from a template, the value of a ValueProvider vp is evaluated either (1) when the template is created or (2) the first time that vp.get() is called in each instantiation of that value provider.

There needs to be a mechanism of specifying that a ValueProvider is evaluated once at the start of the running of the pipeline, and that the value is the same among all instances.  I cannot find any way to do so.

The two obvious examples I can come up with are:

 
```

      ValueProvider startTime;
      ValueProvider shortLivedAccessToken;

```

The obvious rebuttal is that the user could pass \--startTimeMs or \--shortTimeAccessToken as a parameter to the dataflow. 

 
* For the access token, the user may not have the permissions to get this token, and repeatedly requesting a new token is expensive and may hit system request limits.
* For the "start time", the dataflow might be used to perform periodic maintenance in which old entries are deleted.  A bad argument (accidental or malicious) putting startTime in the future could cause the system to think that *everything* is old.  There is no simple mechanism to check the passed parameter for reasonableness.

I can get either of these as a side input, but not as a ValueProvider.

Imported from Jira [BEAM-4295](https://issues.apache.org/jira/browse/BEAM-4295). Original Jira may contain additional context.
Reported by: fyellin.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.