mars-project / mars-project/mars
[BUG] raydataset.to_ray_dataset has type error
@chaokunyang is already working on this.
Since Feb 14, 2022.
- Dominant language
- Python
- Stars
- 2.7k
- Forks
- 325
- PR merge metrics
- No merged PRs in 30d
Description
**Describe the bug**
Got an runtime error when using raydataset.to_ray_dataset
**To Reproduce**
To help us reproducing this bug, please provide information below:
1. Your Python version
3.7.7 from ray 1.9 docker
2. The version of Mars you use
0.8.1
3. Versions of crucial packages, such as numpy, scipy and pandas
4. Full stack of the error.
022-02-13 22:41:24,618 INFO services.py:1340 -- View the Ray dashboard at http://127.0.0.1:8265
Web service started at http://0.0.0.0:42881
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100.0/100 [00:00<00:00, 1044.45it/s]
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100.0/100 [00:00<00:00, 347.11it/s]
a 508.224206
b 493.014249
c 501.428825
d 474.742166
dtype: float64
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100.0/100 [00:05<00:00, 17.29it/s]
a b c d
count 1000.000000 1000.000000 1000.000000 1000.000000
mean 0.508224 0.493014 0.501429 0.474742
std 0.279655 0.286950 0.293181 0.288133
min 0.000215 0.000065 0.000778 0.001233
25% 0.271333 0.238045 0.249944 0.224812
50% 0.516350 0.498089 0.503308 0.459224
75% 0.747174 0.730087 0.750066 0.716232
max 0.999077 0.999674 0.999869 0.999647
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100.0/100 [00:00<00:00, 1155.09it/s]
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100.0/100 [00:00<00:00, 712.39it/s]
Traceback (most recent call last):
File "distributed_mars.py", line 19, in
ds = to_ray_dataset(df, num_shards=4)
File "/home/ray/anaconda3/lib/python3.7/site-packages/mars/dataframe/contrib/raydataset/dataset.py", line 51, in to_ray_dataset
return real_ray_dataset.from_pandas(chunk_refs)
File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/data/read_api.py", line 557, in from_pandas
return from_pandas_refs([ray.put(df) for df in dfs])
File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/data/read_api.py", line 557, in
return from_pandas_refs([ray.put(df) for df in dfs])
File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/_private/client_mode_hook.py", line 105, in wrapper
return func(*args, **kwargs)
File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 1776, in put
value, owner_address=serialize_owner_address)
File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/worker.py", line 283, in put_object
"Calling 'put' on an ray.ObjectRef is not allowed "
TypeError: Calling 'put' on an ray.ObjectRef is not allowed (similarly, returning an ray.ObjectRef from a remote function is not allowed). If you really want to do this, you can wrap the ray.ObjectRef in a list and call 'put' on it (or return it).
Exception ignored in: .cb at 0x7f3f98904ef0>
Traceback (most recent call last):
File "/home/ray/anaconda3/lib/python3.7/site-packages/mars/core/entity/executable.py", line 52, in cb
fut = _decref_pool.submit(decref)
File "/home/ray/anaconda3/lib/python3.7/concurrent/futures/thread.py", line 163, in submit
raise RuntimeError('cannot schedule new futures after shutdown')
RuntimeError: cannot schedule new futures after shutdown
Exception ignored in: .cb at 0x7f3f989045f0>
Traceback (most recent call last):
File "/home/ray/anaconda3/lib/python3.7/site-packages/mars/core/entity/executable.py", line 52, in cb
File "/home/ray/anaconda3/lib/python3.7/concurrent/futures/thread.py", line 163, in submit
RuntimeError: cannot schedule new futures after shutdown
5. Minimized code to reproduce the error.
```python
import ray
ray.init()
import mars
import mars.tensor as mt
import mars.dataframe as md
from mars.dataframe.contrib.raydataset import to_ray_dataset
session = mars.new_ray_session(worker_num=2, worker_mem=1 * 1024 ** 3)
mt.random.RandomState(0).rand(1000, 5).sum().execute()
df = md.DataFrame(
mt.random.rand(1000, 4, chunk_size=500),
columns=list('abcd'))
df.extra_params.raw_chunk_size = 500
print(df.sum().execute())
print(df.describe().execute())
# Convert mars dataframe to ray dataset
df.execute()
ds = to_ray_dataset(df, num_shards=4)
print(ds.schema(), ds.count())
ds.filter(lambda row: row["a"] > 0.5).show(5)
# Convert ray dataset to mars dataframe
df2 = md.read_ray_dataset(ds)
print(df2.head(5).execute())
```
**Expected behavior**
A type error error caused by to_ray_dataset.
" TypeError: Calling 'put' on an ray.ObjectRef is not allowed (similarly, returning an ray.ObjectRef from a remote function is not allowed). If you really want to do this, you can wrap the ray.ObjectRef in a list and call 'put' on it (or return it)."
**Additional context**
Add any other context about the problem here.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.