apache / apache/beam

Spec out what mutations are allowed to a constructed model pipeline, particularly coders

Open
#18,538 0 comments 0 reactions 0 assignees View on GitHub
beam-model improvement P3 portability
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
2d 2h
Merged PRs (30d)
205

Description

Context: presume an SDK has constructed a pipeline or sub-pipeline, and sent it - as a model proto - to another party, which could be a runner or another SDK.

Question to be resolved: What mutations are allowed to this pipeline?

For example, depending on how an SDK harness is implemented, some coders (aka wire formats) can be swapped while leaving the language-level types compatible. For example, "urn:beam:coder:varlong" and "urn:beam:coder:bigendianlong". It may also be possible to add or remove added length prefixes in some situations.

What we mean by _coder_ is a wire format specification for a _stream_ of elements, specified by a `FunctionSpec` proto and its components coders (and so on recursively).

For many coders, if the encoding is not known to a party, then the boundaries of elements cannot be discerned. But there are lots of situations where boundaries need to be known without full decoding - particularly by runners, but also at some point for SDK-to-SDK transmission.

*Possibility 1*: insist that a coder...

```

Coder {
spec: FunctionSpec { urn: "beam:coder:my_whatever_coder" }
}

```

... is always allowed to be replaced by the same coder, wrapped with an added lengh prefix ...

```

Coder {
spec: FunctionSpec { urn: "beam:coder:add_length_prefix" }
component_coders: [
Coder
{
spec: FunctionSpec { urn: "beam:coder:my_whatever_coder" }
}
]
}

```

There is a responsibility that each SDK harness understand this coder and also be able to execute the same UDFs with the decoded values. This is already sort of implicit in how the Fn API produces ProcessBundleDescriptors, since a runner can never assume to understand SDK coders.

*Posibility 2*: allow optimization by indicating a way to determine element boundaries

It may be that even for a coder that cannot be understood, the element boundaries can be easily discerned. For example, if a coder _already_ puts a length prefix in a known format at the start of each element, you just need to pull that out. This means that for an unknown coder, you can save the computation and space of adding a length prefix. (if you can understand "urn:beam:coder:add_length_prefix" then that special case is already handled)

It might look something like this:

```

Coder {
spec: FunctionSpec { urn: "beam:coder:my_whatever_coder" }
also_decodes_as: Coder {

spec: FunctionSpec { urn: "beam:coder:add_length_prefix" }
component_coders: [
Coder:
{ urn: "beam:coder:uninterpretable_bytes" }
]
}
}

```

The extra coder in `also_decodes_as` must be completely wire-compatible and should always be compose of completely standardized coders, so element boundaries can always be ascertained. An annoyance here is the possibility for silly protos where this recurses. Since the main implementation we expect is a length prefix, it could just be a flag, or just a coder for the length prefix itself.

Imported from Jira [BEAM-3203](https://issues.apache.org/jira/browse/BEAM-3203). Original Jira may contain additional context.
Reported by: kenn.

Contributor guide

Open the contributing guide

Research direction

Start with the Fn API model proto, FunctionSpec, coder definitions, and ProcessBundleDescriptor references mentioned in the issue. Establish which coder mutations preserve wire compatibility and element boundaries, then document the allowed rules and how SDK harnesses and runners should determine that the result is valid.

Written by the indexing model from the issue text.

Assessment

Domain
backend-api-design, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.