Pickling error when pickling a nested function
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 2d 5h
- Merged PRs (30d)
- 204
Description
There is a pickling error under the following conditions all happen:
- a function is defined inside a transforms' apply method
- then using it as MapFn
- that function references an instance variable of the outer transform.
Rewriting the nested function as an unnested DoFn appears to solve the problem.
If the limitations of pickling make it difficult to support nested functions then perhaps there's a way to make it easier for users to detect problems caused by nested functions and recommend appropriate fixes
Imported from Jira [BEAM-527](https://issues.apache.org/jira/browse/BEAM-527). Original Jira may contain additional context.
Reported by: altay.
Contributor guide
Research direction
Start by reproducing the nested-function and instance-variable conditions described in the issue, then compare them with the unnested DoFn workaround. Review the imported Jira issue for additional context; done should mean either the case works or users receive clear guidance about the limitation and an appropriate fix.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100