Allow a DoFn to opt in to mutating it's input
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
Runners generally can't tell if a DoFn is mutating inputs, but assuming so by default leads to significant performance implications from unnecessary copying (around sibling fusion, etc). So instead the model prevents mutating inputs, and the Direct Runner validates this behavior. (See: http://beam.incubator.apache.org/contribute/design-principles/#make-efficient-things-easy-rather-than-make-easy-things-efficient)
However, if users are processing a small number of large records by making incremental changes (for example, genomics use cases), the cost of immutability requirement can be very large. As a workaround, users sometimes do suboptimal things (fusing ParDos by hand) or undefined things when they expect the immutability requirement is unnecessarily strict (adding no-op coders in places they hope the runner won't be materializing things, mutating things anyway when they don't expect sibling fusion to happen, etc).
We should consider adding a signal (MutatingDoFn?) that users explicitly opt in to to say their code may mutate inputs. The runner can then use this assumption to either prevent optimizations that would break in the face of this or insert additional copies as needed to allow optimizations to preserve semantics.
See this related user@ discussion:
https://lists.apache.org/thread.html/f39689f54147117f3fc54c498eff1a20fa73f1be5b5cad5b6f816fd3@%3Cuser.beam.apache.org%3E
Imported from Jira [BEAM-1164](https://issues.apache.org/jira/browse/BEAM-1164). Original Jira may contain additional context.
Reported by: frances.
Contributor guide
Research direction
Start with the immutability discussion in this issue, the linked user@ discussion, and the imported Jira BEAM-1164 context. Define how a MutatingDoFn-style opt-in should affect runner optimizations, copying, and Direct Runner validation; done means the behavior and runner semantics are specified well enough to implement.
Written by the indexing model from the issue text.
Assessment
- Domain
- data-engineering, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100