mars-project / mars-project/mars
[BUG]Test on ray cluster: TypeError: __init__() missing 1 required positional argument: 'pid'
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 2.7k
- Forks
- 325
- PR merge metrics
- No merged PRs in 30d
Description
**Describe the bug**
When I run the example code in [mars documentation](https://docs.pymars.org/zh_CN/latest/installation/ray.html), I got the error msg
my code like this
```
import ray
import time
import mars
import mars.tensor as mt
import mars.dataframe as md
def ray_init():
while True:
try:
print(ray.init(address='xx.xx.xx.xx:9999',
log_to_driver=True,
ignore_reinit_error=True,
))
break
except ConnectionError:
print("Ray head is not ready yet, retry")
time.sleep(RAY_WORKER_DELAY)
ray_init()
cluster = mars.new_cluster_in_ray(worker_num=2, worker_mem=2 * 1024 ** 3)
mt.random.RandomState(0).rand(1000_000000, 5).sum().execute()
df = md.DataFrame(
mt.random.rand(1000_0000, 4, chunk_size=500_0000),
columns=list('abcd'))
print(df.sum().execute())
print(df.describe().execute())
# Convert mars dataframe to ray dataset
ds = md.to_ray_dataset(df)
print(ds.schema(), ds.count())
ds.filter(lambda row: row["a"] > 0.5).show(5)
```
**To Reproduce**
To help us reproducing this bug, please provide information below:
1. Your Python version: 3.8.13
2. The version of Mars you use: 0.9.0
3. Versions of crucial packages, such as numpy, scipy and pandas
i. numpy: 1.23.0
ii. ray: 1.13.0
iii. scipy: 1.8.1
iv. pandas: 1.4.2
4. Full stack of the error.
```
(RaySubPool pid=27115) 2022-07-28 10:58:23,066 ERROR serialization.py:342 -- __init__() missing 1 required positional argument: 'pid'
(RaySubPool pid=27115) Traceback (most recent call last):
(RaySubPool pid=27115) File "/home/my_project/anaconda3/envs/my_project/lib/python3.8/site-packages/ray/serialization.py", line 340, in deserialize_objects
(RaySubPool pid=27115) obj = self._deserialize_object(data, metadata, object_ref)
(RaySubPool pid=27115) File "/home/my_project/anaconda3/envs/my_project/lib/python3.8/site-packages/mars/oscar/backends/ray/communication.py", line 169, in _deserialize_object
(RaySubPool pid=27115) value = _ray_deserialize_object(self, data, metadata, object_ref)
(RaySubPool pid=27115) File "/home/my_project/anaconda3/envs/my_project/lib/python3.8/site-packages/ray/serialization.py", line 237, in _deserialize_object
(RaySubPool pid=27115) return self._deserialize_msgpack_data(data, metadata_fields)
(RaySubPool pid=27115) File "/home/my_project/anaconda3/envs/my_project/lib/python3.8/site-packages/ray/serialization.py", line 192, in _deserialize_msgpack_data
(RaySubPool pid=27115) python_objects = self._deserialize_pickle5_data(pickle5_data)
(RaySubPool pid=27115) File "/home/my_project/anaconda3/envs/my_project/lib/python3.8/site-packages/ray/serialization.py", line 182, in _deserialize_pickle5_data
(RaySubPool pid=27115) obj = pickle.loads(in_band)
(RaySubPool pid=27115) File "/home/my_project/anaconda3/envs/my_project/lib/python3.8/site-packages/mars/lib/tblib/pickling_support.py", line 26, in unpickle_exception
(RaySubPool pid=27115) inst = func(*args)
(RaySubPool pid=27115) TypeError: __init__() missing 1 required positional argument: 'pid'
(RaySubPool pid=27115) Get object ObjectRef(a9e08a2346149af818688fe9862d1fe4520f05550900000001000000) from ray://ray-cluster-1658977074/2/0 failed, got exception System error: __init__() missing 1 required positional argument: 'pid'
(RaySubPool pid=27115) traceback: Traceback (most recent call last):
(RaySubPool pid=27115) File "/home/my_project/anaconda3/envs/my_project/lib/python3.8/site-packages/ray/serialization.py", line 340, in deserialize_objects
(RaySubPool pid=27115) obj = self._deserialize_object(data, metadata, object_ref)
(RaySubPool pid=27115) File "/home/my_project/anaconda3/envs/my_project/lib/python3.8/site-packages/mars/oscar/backends/ray/communication.py", line 169, in _deserialize_object
(RaySubPool pid=27115) value = _ray_deserialize_object(self, data, metadata, object_ref)
(RaySubPool pid=27115) File "/home/my_project/anaconda3/envs/my_project/lib/python3.8/site-packages/ray/serialization.py", line 237, in _deserialize_object
(RaySubPool pid=27115) return self._deserialize_msgpack_data(data, metadata_fields)
(RaySubPool pid=27115) File "/home/my_project/anaconda3/envs/my_project/lib/python3.8/site-packages/ray/serialization.py", line 192, in _deserialize_msgpack_data
(RaySubPool pid=27115) python_objects = self._deserialize_pickle5_data(pickle5_data)
(RaySubPool pid=27115) File "/home/my_project/anaconda3/envs/my_project/lib/python3.8/site-packages/ray/serialization.py", line 182, in _deserialize_pickle5_data
(RaySubPool pid=27115) obj = pickle.loads(in_band)
(RaySubPool pid=27115) File "/home/my_project/anaconda3/envs/my_project/lib/python3.8/site-packages/mars/lib/tblib/pickling_support.py", line 26, in unpickle_exception
(RaySubPool pid=27115) inst = func(*args)
(RaySubPool pid=27115) TypeError: __init__() missing 1 required positional argument: 'pid'
(RaySubPool pid=27115) .
(RaySubPool pid=27115) Traceback (most recent call last):
(RaySubPool pid=27115) File "/home/my_project/anaconda3/envs/my_project/lib/python3.8/site-packages/mars/oscar/backends/ray/communication.py", line 285, in handle_task
(RaySubPool pid=27115) result = await object_ref
(RaySubPool pid=27115) ray.exceptions.RaySystemError: System error: __init__() missing 1 required positional argument: 'pid'
(RaySubPool pid=27115) traceback: Traceback (most recent call last):
(RaySubPool pid=27115) File "/home/my_project/anaconda3/envs/my_project/lib/python3.8/site-packages/ray/serialization.py", line 340, in deserialize_objects
(RaySubPool pid=27115) obj = self._deserialize_object(data, metadata, object_ref)
(RaySubPool pid=27115) File "/home/my_project/anaconda3/envs/my_project/lib/python3.8/site-packages/mars/oscar/backends/ray/communication.py", line 169, in _deserialize_object
(RaySubPool pid=27115) value = _ray_deserialize_object(self, data, metadata, object_ref)
(RaySubPool pid=27115) File "/home/my_project/anaconda3/envs/my_project/lib/python3.8/site-packages/ray/serialization.py", line 237, in _deserialize_object
(RaySubPool pid=27115) return self._deserialize_msgpack_data(data, metadata_fields)
(RaySubPool pid=27115) File "/home/my_project/anaconda3/envs/my_project/lib/python3.8/site-packages/ray/serialization.py", line 192, in _deserialize_msgpack_data
(RaySubPool pid=27115) python_objects = self._deserialize_pickle5_data(pickle5_data)
(RaySubPool pid=27115) File "/home/my_project/anaconda3/envs/my_project/lib/python3.8/site-packages/ray/serialization.py", line 182, in _deserialize_pickle5_data
(RaySubPool pid=27115) obj = pickle.loads(in_band)
(RaySubPool pid=27115) File "/home/my_project/anaconda3/envs/my_project/lib/python3.8/site-packages/mars/lib/tblib/pickling_support.py", line 26, in unpickle_exception
(RaySubPool pid=27115) inst = func(*args)
(RaySubPool pid=27115) TypeError: __init__() missing 1 required positional argument: 'pid'
```
5. Minimized code to reproduce the error.
**Expected behavior**
A clear and concise description of what you expected to happen.
**Additional context**
Find #2860 has the similar error msg
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with mars/lib/tblib/pickling_support.py and mars/oscar/backends/ray/communication.py, then compare the similar report in issue #2860. Reproduce the documented Ray-cluster example using the reported Python, Mars, NumPy, SciPy, pandas, and Ray versions; done means the example completes without the deserialization TypeError and the relevant behavior is covered by a regression test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- numpy, pandas, python
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100