cloudpipe / cloudpipe/cloudpickle
TOPIC: submodules usage handling
- Dominant language
- Python
- Stars
- 1.9k
- Forks
- 195
- Avg merge
- 1d 10h
- Merged PRs (30d)
- 1
Description
When pickling a function and depickling it in a new python interpreter (using
cloudpickle of course), all modules used by the function must be imported
before the function is run.
## The current submodule saving strategy has weaknesses
This is actually tricky, because submodules can be referred as attributes to
their parent module. Their usage do not generate any particular opcode that we
can find. As a result, we have no choice but to rely on approximate heuristics
to find submodule usage. These heuristics can be summarized as follows:
(Serializing function f)
```python
for modname, mod in f.__globals__.items():
if is_package(mod):
for submodules in currently_loaded_submodules(mod):
if possible_module_usage(f, submodule)
save(submodule)
```
There are two sources of uncertainty in this loop:
1. figuring out if the submodule is used
2. We are only introspecting `f`'s `__code__` attribute, instead of doing it
recursively
The first bit is not dramatic, as it mostly generates false positives, i.e it
saves modules that should not be saved. The second bit however can make us
miss potential submodule usage, and generate errors in the children-environment runtime.
Here is a fairly legit example were the second point is exploited:
```python
import xml
import xml.etree
import cloudpickle
import sys
def f():
a = xml # generate a LOAD_GLOBAL opcode
def g():
return xml.etree
return g
s = cloudpickle.dumps(f)
# simulate a new, empty children environment
del xml
del sys.modules['xml']
del sys.modules['xml.etree']
cloned_f = cloudpickle.loads(s)
cloned_f()() # raises AttributeError
```
Here, even though `xml` is present in `f`'s globals, because `etree` usage is
hidden in a nested function, it will not be present in `f`'s `co_names`, and
thus not be saved.
In this case, it may sound reasonable to look for code objects in `f`'s
constants, and make subimports saving recursive. However, it is not possible in
all use-cases. Here is a (somewhat crazy) example:
```python
import xml
import xml.etree
import sys
import cloudpickle
class A:
def __new__(cls):
return xml
def f():
a = A()
return a.etree
s = cloudpickle.dumps(f)
del xml
del sys.modules['xml']
del sys.modules['xml.etree']
cloned_f = cloudpickle.loads(s)
cloned_f() # raises AttributeError
```
Here, we cannot start introspecting `A`'s constructor to look for potential
module references.
## Alternatives
On the other side, there exists a range of safe strategies that will save more
modules than necessary, but will not fail at runtime. For example:
* the stupid (but that works) way is simply saving `sys.modules` in its
entirety (Note that saving modules boils down to saving its name as it is
saved by reference). This may be too brutal, and will save hundreds of
unnecessary names when ran from `IPython` shells, as `sys.modules` is
populated with hundreds of modules (606 on my local laptop) at startup time.
* cut the whole module usage inference part: simply look for any name
associated with a `LOAD_GLOBAL` or `LOAD_DEREF`, and save all of its
submodules. This would solve the `IPython` issue while being quasi-bulletproof
(I have not thought of an example breaking it).
Finally, we could add switch like `fast=True` to the `Cloudpickler`, to allow
the user to switch from the safe method current, "unsafe" one. We would
document that enabling fast lead to runtime `AttributeError` errors in extreme
cases.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.