mars-project / mars-project/mars

[BUG] sort_values failed after using dropna

Open
#2,488 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

good first issue Hacktoberfest mod: dataframe pr welcome task: easy type: bug
Dominant language
Python
Stars
2.7k
Forks
325
PR merge metrics
No merged PRs in 30d

Description

**Describe the bug**
when I try to use sort_values(ignore_index=True) after dropna, it raises TypeError:
```
a = md.Series([1,3,2,np.nan,np.nan])
a.dropna().sort_values(ignore_index=True).execute()
```
but I can do it in pandas:
```
b = pd.Series([1, 3, 2, np.nan, np.nan])
b.dropna().sort_values(ignore_index=True)
```

**To Reproduce**
To help us reproducing this bug, please provide information below:
1. Your Python version: 3.8.0
2. The version of Mars you use: 0.6.11
3. Versions of crucial packages, such as numpy, scipy and pandas: pandas: 1.1.3
4. Full stack of the error.
```
ValueError Traceback (most recent call last)
c:\users\hoa'r'jou'r\appdata\local\programs\python\python38\lib\site-packages\pandas\core\dtypes\common.py in ensure_python_int(value)
170 try:
--> 171 new_value = int(value)
172 assert new_value == value

ValueError: cannot convert float NaN to integer

The above exception was the direct cause of the following exception:

TypeError Traceback (most recent call last)
in
1 a = md.Series([1,3,2,np.nan,np.nan])
----> 2 a.dropna().sort_values(ignore_index=True).execute()

c:\users\hoa'r'jou'r\appdata\local\programs\python\python38\lib\site-packages\mars\dataframe\sort\sort_values.py in series_sort_values(series, axis, ascending, inplace, kind, na_position, ignore_index, parallel_kind, psrs_kinds)
317 parallel_kind=parallel_kind, psrs_kinds=psrs_kinds,
318 output_types=[OutputType.series], gpu=series.op.is_gpu())
--> 319 sorted_series = op(series)
320 if inplace:
321 series.data = sorted_series.data

c:\users\hoa'r'jou'r\appdata\local\programs\python\python38\lib\site-packages\mars\utils.py in _inner(*args, **kwargs)
454 def _inner(*args, **kwargs):
455 with self:
--> 456 return func(*args, **kwargs)
457
458 return _inner

c:\users\hoa'r'jou'r\appdata\local\programs\python\python38\lib\site-packages\mars\dataframe\sort\sort_values.py in __call__(self, a)
97 assert self.axis == 0
98 if self.ignore_index:
---> 99 index_value = parse_index(pd.RangeIndex(a.shape[0]))
100 else:
101 if isinstance(a.index_value.value, IndexValue.RangeIndex):

c:\users\hoa'r'jou'r\appdata\local\programs\python\python38\lib\site-packages\pandas\core\indexes\range.py in __new__(cls, start, stop, step, dtype, copy, name)
100 raise TypeError("RangeIndex(...) must be called with integers")
101
--> 102 start = ensure_python_int(start) if start is not None else 0
103
104 if stop is None:

c:\users\hoa'r'jou'r\appdata\local\programs\python\python38\lib\site-packages\pandas\core\dtypes\common.py in ensure_python_int(value)
172 assert new_value == value
173 except (TypeError, ValueError, AssertionError) as err:
--> 174 raise TypeError(f"Wrong type {type(value)} for value {value}") from err
175 return new_value
176

TypeError: Wrong type for value nan
```

**Expected behavior**
A clear and concise description of what you expected to happen.

**Additional context**
Add any other context about the problem here.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in mars/dataframe/sort/sort_values.py, at the Series sort_values path that builds a RangeIndex when ignore_index is true. Reproduce the reported dropna().sort_values(ignore_index=True) case and inspect the index shape or value passed to pandas. Done means the operation executes without TypeError and produces the expected reset index, with a regression test for the example.

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas, python
Domain
data
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.