mars-project / mars-project/mars

A question about how to read a csv file and output it

Open
#3,232 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
2.7k
Forks
325
PR merge metrics
No merged PRs in 30d

Description

Hi!
My environment are centos6.7, python3.7, and pymars0.9.0. When I run the demo you give in zhihu [https://www.zhihu.com/question/320961999](url) , something seems to be wrong.
Here are my codes. I think maybe 'execute()' leads to the problem.

```
import mars.dataframe as md

df = md.read_csv('bin_end.csv')

df.execute()
```

Here are the error logs.

```
/usr/local/lib/python3.7/site-packages/mars/deploy/oscar/session.py:2064: UserWarning:
No session found, local session will be created in background, it may take a while before execution. If you want to new a local session by yourself, run code below:

import mars

mars.new_session()

warnings.warn(warning_msg)
Traceback (most recent call last):
File "my.py", line 5, in
df.execute()
File "/usr/local/lib/python3.7/site-packages/mars/core/entity/tileables.py", line 462, in execute
result = self.data.execute(session=session, **kw)
File "/usr/local/lib/python3.7/site-packages/mars/core/entity/executable.py", line 144, in execute
return execute(self, session=session, **kw)
File "/usr/local/lib/python3.7/site-packages/mars/deploy/oscar/session.py", line 1888, in execute
session = get_default_or_create(**(new_session_kwargs or dict()))
File "/usr/local/lib/python3.7/site-packages/mars/deploy/oscar/session.py", line 2065, in get_default_or_create
session = new_session("127.0.0.1", init_local=True, **kwargs)
File "/usr/local/lib/python3.7/site-packages/mars/deploy/oscar/session.py", line 2040, in new_session
address, session_id=session_id, backend=backend, new=new, **kwargs
File "/usr/local/lib/python3.7/site-packages/mars/deploy/oscar/session.py", line 1636, in init
isolated_session = fut.result()
File "/usr/local/lib/python3.7/concurrent/futures/_base.py", line 432, in result
return self.__get_result()
File "/usr/local/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result
raise self._exception
File "/usr/local/lib/python3.7/site-packages/mars/deploy/oscar/session.py", line 848, in init
address, timeout=timeout, backend=backend, **kwargs
File "/usr/local/lib/python3.7/site-packages/mars/deploy/oscar/local.py", line 91, in new_cluster_in_isolation
await cluster.start()
File "/usr/local/lib/python3.7/site-packages/mars/deploy/oscar/local.py", line 219, in start
await self._start_service()
File "/usr/local/lib/python3.7/site-packages/mars/deploy/oscar/local.py", line 274, in _start_service
config=self._config,
File "/usr/local/lib/python3.7/site-packages/mars/deploy/oscar/service.py", line 84, in start_worker
NodeRole.WORKER, config, address=address, mark_ready=mark_ready
File "/usr/local/lib/python3.7/site-packages/mars/services/core.py", line 174, in start_services
await asyncio.gather(*[inst.start() for inst in instances])
File "/usr/local/lib/python3.7/site-packages/mars/services/cluster/worker/service.py", line 68, in start
address=address,
File "/usr/local/lib/python3.7/site-packages/mars/oscar/api.py", line 27, in create_actor
return await ctx.create_actor(actor_cls, *args, uid=uid, address=address, **kwargs)
File "/usr/local/lib/python3.7/site-packages/mars/oscar/backends/context.py", line 112, in create_actor
return self._process_result_message(result)
File "/usr/local/lib/python3.7/site-packages/mars/oscar/backends/context.py", line 76, in _process_result_message
raise message.as_instanceof_cause()
File "/usr/local/lib/python3.7/site-packages/mars/oscar/backends/pool.py", line 525, in create_actor
await self._run_coro(message.message_id, actor.__post_create__())
File "/usr/local/lib/python3.7/site-packages/mars/oscar/backends/pool.py", line 343, in _run_coro
return await coro
File "/usr/local/lib/python3.7/site-packages/mars/services/cluster/procinfo.py", line 41, in __post_create__
allocate_strategy=ProcessIndex(index),
File "/usr/local/lib/python3.7/site-packages/mars/oscar/api.py", line 27, in create_actor
return await ctx.create_actor(actor_cls, *args, uid=uid, address=address, **kwargs)
File "/usr/local/lib/python3.7/site-packages/mars/oscar/backends/context.py", line 112, in create_actor
return self._process_result_message(result)
File "/usr/local/lib/python3.7/site-packages/mars/oscar/backends/context.py", line 76, in _process_result_message
raise message.as_instanceof_cause()
File "/usr/local/lib/python3.7/site-packages/mars/oscar/backends/pool.py", line 939, in create_actor
result = await self.call(address, new_create_actor_message)
File "/usr/local/lib/python3.7/site-packages/mars/oscar/backends/pool.py", line 402, in call
return await self._caller.call(self._router, dest_address, message)
File "/usr/local/lib/python3.7/site-packages/mars/oscar/backends/core.py", line 95, in call
await client.send(message)
File "/usr/local/lib/python3.7/site-packages/mars/oscar/backends/communication/base.py", line 258, in send
return await self.channel.send(message)
File "/usr/local/lib/python3.7/site-packages/mars/oscar/backends/communication/socket.py", line 73, in send
buffers = await serializer.run()
File "/usr/local/lib/python3.7/site-packages/mars/serialization/aio.py", line 85, in run
return await self._get_buffers()
File "/usr/local/lib/python3.7/site-packages/mars/serialization/aio.py", line 41, in _get_buffers
self._obj, spawn_threshold=DEFAULT_SPAWN_THRESHOLD
File "mars/serialization/core.pyx", line 766, in serialize_with_spawn
File "mars/serialization/core.pyx", line 660, in mars.serialization.core._serialize_with_stack
File "mars/serialization/core.pyx", line 598, in mars.serialization.core._serial_single
File "mars/serialization/core.pyx", line 258, in mars.serialization.core.PickleSerializer.serial
File "mars/serialization/core.pyx", line 223, in mars.serialization.core.pickle_buffers
File "/usr/local/lib/python3.7/site-packages/cloudpickle/cloudpickle_fast.py", line 73, in dumps
cp.dump(obj)
File "/usr/local/lib/python3.7/site-packages/cloudpickle/cloudpickle_fast.py", line 640, in dump
raise pickle.PicklingError(msg) from e
pickle5.pickle.PicklingError: [address=127.0.0.1:33346, pid=30517] Could not pickle object as excessively deep recursion required.
```

Contributor guide

Open the contributing guide

Research direction

Reproduce the provided read_csv and execute() example in the stated Python, CentOS, and Mars environment, then start with the traceback entries in deploy/oscar/session.py, deploy/oscar/local.py, and serialization/core.pyx. Done means the example executes without the reported excessively deep recursion PicklingError and produces the requested CSV output.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.