cloudpipe / cloudpipe/cloudpickle

TOPIC: submodules usage handling

Open
#255 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.9k
Forks
195
Avg merge
1d 10h
Merged PRs (30d)
1

Description

When pickling a function and depickling it in a new python interpreter (using
cloudpickle of course), all modules used by the function must be imported
before the function is run.

## The current submodule saving strategy has weaknesses

This is actually tricky, because submodules can be referred as attributes to
their parent module. Their usage do not generate any particular opcode that we
can find. As a result, we have no choice but to rely on approximate heuristics
to find submodule usage. These heuristics can be summarized as follows:

(Serializing function f)
```python
for modname, mod in f.__globals__.items():
if is_package(mod):
for submodules in currently_loaded_submodules(mod):
if possible_module_usage(f, submodule)
save(submodule)
```

There are two sources of uncertainty in this loop:
1. figuring out if the submodule is used
2. We are only introspecting `f`'s `__code__` attribute, instead of doing it
recursively

The first bit is not dramatic, as it mostly generates false positives, i.e it
saves modules that should not be saved. The second bit however can make us
miss potential submodule usage, and generate errors in the children-environment runtime.

Here is a fairly legit example were the second point is exploited:

```python
import xml
import xml.etree
import cloudpickle
import sys

def f():
a = xml # generate a LOAD_GLOBAL opcode
def g():
return xml.etree
return g
s = cloudpickle.dumps(f)

# simulate a new, empty children environment
del xml
del sys.modules['xml']
del sys.modules['xml.etree']

cloned_f = cloudpickle.loads(s)
cloned_f()() # raises AttributeError
```

Here, even though `xml` is present in `f`'s globals, because `etree` usage is
hidden in a nested function, it will not be present in `f`'s `co_names`, and
thus not be saved.

In this case, it may sound reasonable to look for code objects in `f`'s
constants, and make subimports saving recursive. However, it is not possible in
all use-cases. Here is a (somewhat crazy) example:

```python
import xml
import xml.etree
import sys

import cloudpickle

class A:
def __new__(cls):
return xml

def f():
a = A()
return a.etree

s = cloudpickle.dumps(f)

del xml
del sys.modules['xml']
del sys.modules['xml.etree']

cloned_f = cloudpickle.loads(s)
cloned_f() # raises AttributeError
```

Here, we cannot start introspecting `A`'s constructor to look for potential
module references.

## Alternatives

On the other side, there exists a range of safe strategies that will save more
modules than necessary, but will not fail at runtime. For example:

* the stupid (but that works) way is simply saving `sys.modules` in its
entirety (Note that saving modules boils down to saving its name as it is
saved by reference). This may be too brutal, and will save hundreds of
unnecessary names when ran from `IPython` shells, as `sys.modules` is
populated with hundreds of modules (606 on my local laptop) at startup time.
* cut the whole module usage inference part: simply look for any name
associated with a `LOAD_GLOBAL` or `LOAD_DEREF`, and save all of its
submodules. This would solve the `IPython` issue while being quasi-bulletproof
(I have not thought of an example breaking it).

Finally, we could add switch like `fast=True` to the `Cloudpickler`, to allow
the user to switch from the safe method current, "unsafe" one. We would
document that enabling fast lead to runtime `AttributeError` errors in extreme
cases.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.