cloudpipe / cloudpipe/cloudpickle

unexpectly import my local module when it is not necessary

Open
#500 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.9k
Forks
195
Avg merge
1d 10h
Merged PRs (30d)
1

Description

In file ai2_kit/domain.py

```python
def fun(ctx):
def in_add(a, b):
print (a+b)

ctx.executor.run_python_fn(in_add)(1, 2) # this pass
ctx.executor.run_python_fn(out_add)(1, 2) # this failed, the error is: ModuleNotFoundError: No module named 'ai2_kit'

def out_add(a, b):
print(a+b)
```
the method `run_python_fn` is defined in ai2_kit/executor.py , the basic idea is use `python -c` to execute a python script on remote machine.

```python
def run_python_script(self, script: str):
return self.connector.run('python -c {}'.format(shlex.quote(script)))

def run_python_fn(self, fn: T, python_cmd=None) -> T:
def remote_fn(*args, **kwargs):
dumped_fn = base64.b64encode(cloudpickle.dumps(lambda: fn(*args, **kwargs), protocol=pickle.DEFAULT_PROTOCOL))
script = '''import base64,pickle; pickle.loads(base64.b64decode({}))()'''.format(repr(dumped_fn))
self.run_python_script(script=script, python_cmd=python_cmd)
```
I have no idea why it will import `ai2_kit` when use a function outside of current function, the method `out_add` doesn't have any external dependencies. Is there any method to workaround this problem? Thank you! Both local and remote python is v3.9.

The error pickle (base64) for debug:
```
gASVFAMAAAAAAACMF2Nsb3VkcGlja2xlLmNsb3VkcGlja2xllIwNX2J1aWx0aW5fdHlwZZSTlIwKTGFtYmRhVHlwZZSFlFKUKGgCjAhDb2RlVHlwZZSFlFKUKEsASwBLAEsASwRLE0MOiAGIAGkAiAKkAY4BUwCUToWUKSmMRS9ob21lL2hlbnJ5L3NyYy9jYy1wcm9qZWN0cy9wcm9qZWN0cy9haTIta2l0L2FpMl9raXQvY29yZS9leGVjdXRvci5weZSMCDxsYW1iZGE+lEuMQwCUjARhcmdzlIwCZm6UjAZrd2FyZ3OUh5QpdJRSlH2UKIwLX19wYWNrYWdlX1+UjAxhaTJfa2l0LmNvcmWUjAhfX25hbWVfX5SMFWFpMl9raXQuY29yZS5leGVjdXRvcpSMCF9fZmlsZV9flIxFL2hvbWUvaGVucnkvc3JjL2NjLXByb2plY3RzL3Byb2plY3RzL2FpMi1raXQvYWkyX2tpdC9jb3JlL2V4ZWN1dG9yLnB5lHVOTmgAjBBfbWFrZV9lbXB0eV9jZWxslJOUKVKUaBwpUpRoHClSlIeUdJRSlIwcY2xvdWRwaWNrbGUuY2xvdWRwaWNrbGVfZmFzdJSMEl9mdW5jdGlvbl9zZXRzdGF0ZZSTlGgifZR9lChoF2gMjAxfX3F1YWxuYW1lX1+UjD5IcGNFeGVjdXRvci5ydW5fcHl0aG9uX2ZuLjxsb2NhbHM+LnJlbW90ZV9mbi48bG9jYWxzPi48bGFtYmRhPpSMD19fYW5ub3RhdGlvbnNfX5R9lIwOX19rd2RlZmF1bHRzX1+UTowMX19kZWZhdWx0c19flE6MCl9fbW9kdWxlX1+UaBiMB19fZG9jX1+UTowLX19jbG9zdXJlX1+UaACMCl9tYWtlX2NlbGyUk5RLAUsChpSFlFKUaDKME2FpMl9raXQuZG9tYWluLmNwMmuUjANhZGSUk5SFlFKUaDJ9lIWUUpSHlIwXX2Nsb3VkcGlja2xlX3N1Ym1vZHVsZXOUXZSMC19fZ2xvYmFsc19flH2UdYaUhlIwLg==
```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.