DoFns should be serialized at apply time and deserialized when executing
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
1. Serializing DoFns at application time ensures that any modifications of fields within the DoFn after application do not accidentally pollute the execution. This mirrors the approach taken in Java to provide an approximation of lexical-closure (eg., you only need to know the state of the DoFn at the time it was applied, not afterwards, to understand its behavior).
2. Based on 1, the DIrectRunner should also be deserializing DoFns before running them, which should also detect other classes of errors such as using the pipeline object (which is not pickleable) within the DoFn
Imported from Jira [BEAM-681](https://issues.apache.org/jira/browse/BEAM-681). Original Jira may contain additional context.
Reported by: bchambers.
Contributor guide
Research direction
Start by tracing the Python DirectRunner's DoFn application and execution path, using the Java approach mentioned in the issue as a behavioral reference. Done means DoFns are serialized at apply time, deserialized before DirectRunner execution, and errors from non-pickleable pipeline references are detected.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering, distributed-systems
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100