apache / apache/beam

Per-step, per-execution nonce

Open
#18,054 0 comments 0 reactions 0 assignees View on GitHub
core improvement java P3
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
2d 2h
Merged PRs (30d)
205

Description

In the forthcoming runner API, a user will be able to save a pipeline to JSON and then run it repeatedly.

Many pieces of code (e.g., BigQueryIO.Read or Write) rely on a single random value (nonce). These values are typically generated at apply time, so that they are deterministic (don't change across retries of DoFns) and global (are the same across all workers).

However, once the runner API lands the existing code would result in the same nonce being reused across jobs. Other possible solutions:

* Generate nonce in `Create(1) | ParDo` then use this as a side input. Should work, as along as side inputs are actually checkpointed. But does not work for `BoundedSource`.

* If a nonce is only needed for the lifetime of one bundle, can be generated in `startBundle` and used in `finishBundle` [or `tearDown`].

* Add some context somewhere that lets user code access unique step name, and somehow generate a nonce consistently e.g. by hashing. Will usually work, but this is similarly not available to sources.

Another Q: I'm not sure we have a good way to generate nonces in unbounded pipelines \-- we probably need one. This would enable us to, e.g., use `BigQueryIO.Write` in an unbounded pipeline [if we had, e.g., exactly-once triggering per window]. Or generalizing to multiple firings...

Imported from Jira [BEAM-758](https://issues.apache.org/jira/browse/BEAM-758). Original Jira may contain additional context.
Reported by: dhalperi.

Contributor guide

Open the contributing guide

Research direction

Start with the runner API context and the imported Jira BEAM-758, then trace the nonce use cases named in BigQueryIO.Read, BigQueryIO.Write, and BoundedSource. The issue is done only when an agreed approach supports per-step, per-execution nonces across repeated jobs, bounded sources, and unbounded pipelines.

Written by the indexing model from the issue text.

Assessment

Domain
data-engineering, distributed-systems, stream-processing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.