mars-project / mars-project/mars

[BUG]Test on ray cluster: TypeError: __init__() missing 1 required positional argument: 'pid'

Open
#3,197 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
2.7k
Forks
325
PR merge metrics
No merged PRs in 30d

Description

**Describe the bug**
When I run the example code in [mars documentation](https://docs.pymars.org/zh_CN/latest/installation/ray.html), I got the error msg
my code like this

```
import ray
import time

import mars
import mars.tensor as mt
import mars.dataframe as md

def ray_init():
while True:
try:
print(ray.init(address='xx.xx.xx.xx:9999',
log_to_driver=True,
ignore_reinit_error=True,
))
break
except ConnectionError:
print("Ray head is not ready yet, retry")
time.sleep(RAY_WORKER_DELAY)

ray_init()
cluster = mars.new_cluster_in_ray(worker_num=2, worker_mem=2 * 1024 ** 3)
mt.random.RandomState(0).rand(1000_000000, 5).sum().execute()
df = md.DataFrame(
mt.random.rand(1000_0000, 4, chunk_size=500_0000),
columns=list('abcd'))
print(df.sum().execute())
print(df.describe().execute())
# Convert mars dataframe to ray dataset
ds = md.to_ray_dataset(df)
print(ds.schema(), ds.count())
ds.filter(lambda row: row["a"] > 0.5).show(5)
```

**To Reproduce**
To help us reproducing this bug, please provide information below:
1. Your Python version: 3.8.13
2. The version of Mars you use: 0.9.0
3. Versions of crucial packages, such as numpy, scipy and pandas
i. numpy: 1.23.0
ii. ray: 1.13.0
iii. scipy: 1.8.1
iv. pandas: 1.4.2
4. Full stack of the error.
```
(RaySubPool pid=27115) 2022-07-28 10:58:23,066 ERROR serialization.py:342 -- __init__() missing 1 required positional argument: 'pid'
(RaySubPool pid=27115) Traceback (most recent call last):
(RaySubPool pid=27115) File "/home/my_project/anaconda3/envs/my_project/lib/python3.8/site-packages/ray/serialization.py", line 340, in deserialize_objects
(RaySubPool pid=27115) obj = self._deserialize_object(data, metadata, object_ref)
(RaySubPool pid=27115) File "/home/my_project/anaconda3/envs/my_project/lib/python3.8/site-packages/mars/oscar/backends/ray/communication.py", line 169, in _deserialize_object
(RaySubPool pid=27115) value = _ray_deserialize_object(self, data, metadata, object_ref)
(RaySubPool pid=27115) File "/home/my_project/anaconda3/envs/my_project/lib/python3.8/site-packages/ray/serialization.py", line 237, in _deserialize_object
(RaySubPool pid=27115) return self._deserialize_msgpack_data(data, metadata_fields)
(RaySubPool pid=27115) File "/home/my_project/anaconda3/envs/my_project/lib/python3.8/site-packages/ray/serialization.py", line 192, in _deserialize_msgpack_data
(RaySubPool pid=27115) python_objects = self._deserialize_pickle5_data(pickle5_data)
(RaySubPool pid=27115) File "/home/my_project/anaconda3/envs/my_project/lib/python3.8/site-packages/ray/serialization.py", line 182, in _deserialize_pickle5_data
(RaySubPool pid=27115) obj = pickle.loads(in_band)
(RaySubPool pid=27115) File "/home/my_project/anaconda3/envs/my_project/lib/python3.8/site-packages/mars/lib/tblib/pickling_support.py", line 26, in unpickle_exception
(RaySubPool pid=27115) inst = func(*args)
(RaySubPool pid=27115) TypeError: __init__() missing 1 required positional argument: 'pid'
(RaySubPool pid=27115) Get object ObjectRef(a9e08a2346149af818688fe9862d1fe4520f05550900000001000000) from ray://ray-cluster-1658977074/2/0 failed, got exception System error: __init__() missing 1 required positional argument: 'pid'
(RaySubPool pid=27115) traceback: Traceback (most recent call last):
(RaySubPool pid=27115) File "/home/my_project/anaconda3/envs/my_project/lib/python3.8/site-packages/ray/serialization.py", line 340, in deserialize_objects
(RaySubPool pid=27115) obj = self._deserialize_object(data, metadata, object_ref)
(RaySubPool pid=27115) File "/home/my_project/anaconda3/envs/my_project/lib/python3.8/site-packages/mars/oscar/backends/ray/communication.py", line 169, in _deserialize_object
(RaySubPool pid=27115) value = _ray_deserialize_object(self, data, metadata, object_ref)
(RaySubPool pid=27115) File "/home/my_project/anaconda3/envs/my_project/lib/python3.8/site-packages/ray/serialization.py", line 237, in _deserialize_object
(RaySubPool pid=27115) return self._deserialize_msgpack_data(data, metadata_fields)
(RaySubPool pid=27115) File "/home/my_project/anaconda3/envs/my_project/lib/python3.8/site-packages/ray/serialization.py", line 192, in _deserialize_msgpack_data
(RaySubPool pid=27115) python_objects = self._deserialize_pickle5_data(pickle5_data)
(RaySubPool pid=27115) File "/home/my_project/anaconda3/envs/my_project/lib/python3.8/site-packages/ray/serialization.py", line 182, in _deserialize_pickle5_data
(RaySubPool pid=27115) obj = pickle.loads(in_band)
(RaySubPool pid=27115) File "/home/my_project/anaconda3/envs/my_project/lib/python3.8/site-packages/mars/lib/tblib/pickling_support.py", line 26, in unpickle_exception
(RaySubPool pid=27115) inst = func(*args)
(RaySubPool pid=27115) TypeError: __init__() missing 1 required positional argument: 'pid'
(RaySubPool pid=27115) .
(RaySubPool pid=27115) Traceback (most recent call last):
(RaySubPool pid=27115) File "/home/my_project/anaconda3/envs/my_project/lib/python3.8/site-packages/mars/oscar/backends/ray/communication.py", line 285, in handle_task
(RaySubPool pid=27115) result = await object_ref
(RaySubPool pid=27115) ray.exceptions.RaySystemError: System error: __init__() missing 1 required positional argument: 'pid'
(RaySubPool pid=27115) traceback: Traceback (most recent call last):
(RaySubPool pid=27115) File "/home/my_project/anaconda3/envs/my_project/lib/python3.8/site-packages/ray/serialization.py", line 340, in deserialize_objects
(RaySubPool pid=27115) obj = self._deserialize_object(data, metadata, object_ref)
(RaySubPool pid=27115) File "/home/my_project/anaconda3/envs/my_project/lib/python3.8/site-packages/mars/oscar/backends/ray/communication.py", line 169, in _deserialize_object
(RaySubPool pid=27115) value = _ray_deserialize_object(self, data, metadata, object_ref)
(RaySubPool pid=27115) File "/home/my_project/anaconda3/envs/my_project/lib/python3.8/site-packages/ray/serialization.py", line 237, in _deserialize_object
(RaySubPool pid=27115) return self._deserialize_msgpack_data(data, metadata_fields)
(RaySubPool pid=27115) File "/home/my_project/anaconda3/envs/my_project/lib/python3.8/site-packages/ray/serialization.py", line 192, in _deserialize_msgpack_data
(RaySubPool pid=27115) python_objects = self._deserialize_pickle5_data(pickle5_data)
(RaySubPool pid=27115) File "/home/my_project/anaconda3/envs/my_project/lib/python3.8/site-packages/ray/serialization.py", line 182, in _deserialize_pickle5_data
(RaySubPool pid=27115) obj = pickle.loads(in_band)
(RaySubPool pid=27115) File "/home/my_project/anaconda3/envs/my_project/lib/python3.8/site-packages/mars/lib/tblib/pickling_support.py", line 26, in unpickle_exception
(RaySubPool pid=27115) inst = func(*args)
(RaySubPool pid=27115) TypeError: __init__() missing 1 required positional argument: 'pid'

```
5. Minimized code to reproduce the error.

**Expected behavior**
A clear and concise description of what you expected to happen.

**Additional context**
Find #2860 has the similar error msg

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with mars/lib/tblib/pickling_support.py and mars/oscar/backends/ray/communication.py, then compare the similar report in issue #2860. Reproduce the documented Ray-cluster example using the reported Python, Mars, NumPy, SciPy, pandas, and Ray versions; done means the example completes without the deserialization TypeError and the relevant behavior is covered by a regression test.

Written by the indexing model from the issue text.

Assessment

Tech stack
numpy, pandas, python
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.