apache / apache/beam

Pickling error when pickling a nested function

Open
#18,042 0 comments 0 reactions 0 assignees View on GitHub
bug core P3 python
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
2d 5h
Merged PRs (30d)
204

Description

There is a pickling error under the following conditions all happen:
- a function is defined inside a transforms' apply method
- then using it as MapFn
- that function references an instance variable of the outer transform.

Rewriting the nested function as an unnested DoFn appears to solve the problem.

If the limitations of pickling make it difficult to support nested functions then perhaps there's a way to make it easier for users to detect problems caused by nested functions and recommend appropriate fixes

Imported from Jira [BEAM-527](https://issues.apache.org/jira/browse/BEAM-527). Original Jira may contain additional context.
Reported by: altay.

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the nested-function and instance-variable conditions described in the issue, then compare them with the unnested DoFn workaround. Review the imported Jira issue for additional context; done should mean either the case works or users receive clear guidance about the limitation and an appropriate fix.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.