mars-project / mars-project/mars
[BUG] ArrowArray failed with error
Open
Nobody has claimed this yet.
mod: dataframe
type: bug
- Dominant language
- Python
- Stars
- 2.7k
- Forks
- 325
- PR merge metrics
- No merged PRs in 30d
Description
Describe the bug
ArrowArray failed with error.
2020-12-05 22:16:43,592 mars.scheduler.operands.common 57 ERROR Attempt 4: Unexpected error TypeError occurred in executing operand 8d11327f36b6ad13ad323d4719ac9104 in 33.13.158.99:13525
Traceback (most recent call last):
File "/home/admin/work/public-mars-0.5.2.zip/mars/promise.py", line 100, in _wrapped
result = func(*args, **kwargs)
File "/home/admin/work/public-mars-0.5.2.zip/mars/worker/calc.py", line 301, in <lambda>
.then(lambda context_dict: _start_calc(context_dict)) \
File "/home/admin/work/public-mars-0.5.2.zip/mars/worker/calc.py", line 276, in _start_calc
return self._calc_results(session_id, graph_key, graph, context_dict, chunk_targets)
File "/home/admin/work/public-mars-0.5.2.zip/mars/utils.py", line 377, in _wrapped
return func(*args, **kwargs)
File "/home/admin/work/public-mars-0.5.2.zip/mars/worker/calc.py", line 200, in _calc_results
chunk_targets, retval=False).result()
File "src/gevent/event.py", line 383, in gevent._gevent_cevent.AsyncResult.result
File "src/gevent/event.py", line 305, in gevent._gevent_cevent.AsyncResult.get
File "src/gevent/event.py", line 335, in gevent._gevent_cevent.AsyncResult.get
File "src/gevent/event.py", line 323, in gevent._gevent_cevent.AsyncResult.get
File "src/gevent/event.py", line 303, in gevent._gevent_cevent.AsyncResult._raise_exception
File "/opt/conda/lib/python3.7/site-packages/gevent/_compat.py", line 65, in reraise
raise value.with_traceback(tb)
File "/opt/conda/lib/python3.7/site-packages/gevent/threadpool.py", line 142, in __run_task
thread_result.set(func(*args, **kwargs))
File "mars/actors/pool/gevent_pool.pyx", line 127, in mars.actors.pool.gevent_pool.GeventThreadPool._wrap_watch.inner
result = fn(*args, **kwargs)
File "/home/admin/work/public-mars-0.5.2.zip/mars/executor.py", line 690, in execute_graph
res = graph_execution.execute(retval)
File "/home/admin/work/public-mars-0.5.2.zip/mars/executor.py", line 571, in execute
future.result()
File "/opt/conda/lib/python3.7/concurrent/futures/_base.py", line 435, in result
return self.__get_result()
File "/opt/conda/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result
raise self._exception
File "/opt/conda/lib/python3.7/concurrent/futures/thread.py", line 57, in run
result = self.fn(*self.args, **self.kwargs)
File "/home/admin/work/public-mars-0.5.2.zip/mars/utils.py", line 451, in _inner
return func(*args, **kwargs)
File "/home/admin/work/public-mars-0.5.2.zip/mars/executor.py", line 443, in _execute_operand
Executor.handle(first_op, results, self._mock)
File "/home/admin/work/public-mars-0.5.2.zip/mars/executor.py", line 641, in handle
return runner(results, op)
File "/home/admin/work/public-pyodps-0.10.1.zip/odps/mars_extension/dataframe/datastore.py", line 243, in execute
odps_schema = pd_to_df_schema(to_store_data, unknown_as_string=op.unknown_as_string)
File "/home/admin/work/public-pyodps-0.10.1.zip/odps/df/backends/pd/types.py", line 92, in pd_to_df_schema
name=names[i]))
File "/home/admin/work/public-pyodps-0.10.1.zip/odps/df/backends/pd/types.py", line 54, in np_type_to_df_type
for it in arr:
File "/opt/conda/lib/python3.7/site-packages/pandas/core/arrays/base.py", line 352, in __iter__
yield self[i]
File "/home/admin/work/public-mars-0.5.2.zip/mars/dataframe/arrays.py", line 386, in __getitem__
return cls(array[item], dtype=self._dtype)
File "/home/admin/work/public-mars-0.5.2.zip/mars/dataframe/arrays.py", line 538, in __init__
ArrowArray.__init__(self, values, ArrowStringDtype(), copy=copy)
File "/home/admin/work/public-mars-0.5.2.zip/mars/dataframe/arrays.py", line 204, in __init__
self._init_by_arrow(values, dtype=dtype, copy=copy)
File "/home/admin/work/public-mars-0.5.2.zip/mars/dataframe/arrays.py", line 234, in _init_by_arrow
arrow_array = pa.chunked_array([pa.array(values, type=dtype.arrow_type)])
File "pyarrow/array.pxi", line 211, in pyarrow.lib.array
return _sequence_to_array(obj, mask, size, type, pool, c_from_pandas)
File "pyarrow/array.pxi", line 36, in pyarrow.lib._sequence_to_array
check_status(ConvertPySequence(sequence, mask, options, &out))
TypeError: 'NoneType' object is not iterable
To Reproduce
To help us reproducing this bug, please provide information below:
- Your Python version
- The version of Mars you use
- Versions of crucial packages, such as numpy, scipy and protobuf
- Full stack of the error.
- Minimized code to reproduce the error.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with mars/dataframe/arrays.py, especially ArrowArray._init_by_arrow and the ArrowStringArray constructor shown in the traceback. Trace the None value reaching pyarrow.array, then create a minimal reproduction from the reported stack and verify that the failing ArrowArray path no longer raises the TypeError.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- pandas, python
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100