NVIDIA / NVIDIA/cudf

[BUG] fillna with dataframe containing list dtypes fails

Open
#12,274 2 comments 0 reactions 0 assignees View on GitHub
0 - Backlog bug Python
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

**Describe the bug**

fillna with dataframe containing list dtypes fails .

**Steps/Code to reproduce bug**
```python3
import cudf
s1 = cudf.Series(['game', 'game', None,None,None])
s2 = cudf.Series([[1], [1], None,None,None])
df = cudf.DataFrame({'_TYPE_':s1, 'h':s2})

c1 = cudf.Series([None,None, 'user', 'user', 'user'])
c2 = cudf.Series([None, None, [2], [2], [2]])
subdf = cudf.DataFrame({'_TYPE_':c1, 'h':c2})
df.fillna(subdf)
```

```python3
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
Cell In [23], line 9
7 c2 = cudf.Series([None, None, [2], [2], [2]])
8 subdf = cudf.DataFrame({'_TYPE_':c1, 'h':c2})
----> 9 df.fillna(subdf)

File /datasets/vjawa/miniconda3/envs/cugraph_dgl_dev/lib/python3.9/contextlib.py:79, in ContextDecorator.__call__..inner(*args, **kwds)
76 @wraps(func)
77 def inner(*args, **kwds):
78 with self._recreate_cm():
---> 79 return func(*args, **kwds)

File /datasets/vjawa/miniconda3/envs/cugraph_dgl_dev/lib/python3.9/site-packages/cudf/core/indexed_frame.py:1842, in IndexedFrame.fillna(self, value, method, axis, inplace, limit)
1837 @_cudf_nvtx_annotate
1838 def fillna(
1839 self, value=None, method=None, axis=None, inplace=False, limit=None
1840 ): # noqa: D102
1841 old_index = self._index
-> 1842 ret = super().fillna(value, method, axis, inplace, limit)
1843 if inplace:
1844 self._index = old_index

File /datasets/vjawa/miniconda3/envs/cugraph_dgl_dev/lib/python3.9/contextlib.py:79, in ContextDecorator.__call__..inner(*args, **kwds)
76 @wraps(func)
77 def inner(*args, **kwds):
78 with self._recreate_cm():
---> 79 return func(*args, **kwds)

File /datasets/vjawa/miniconda3/envs/cugraph_dgl_dev/lib/python3.9/site-packages/cudf/core/frame.py:905, in Frame.fillna(self, value, method, axis, inplace, limit)
899 should_fill = (
900 col_name in value
901 and col.contains_na_entries
902 and not libcudf.scalar._is_null_host_scalar(replace_val)
903 ) or method is not None
904 if should_fill:
--> 905 filled_data[col_name] = col.fillna(replace_val, method)
906 else:
907 filled_data[col_name] = col.copy(deep=True)

File /datasets/vjawa/miniconda3/envs/cugraph_dgl_dev/lib/python3.9/site-packages/cudf/core/column/column.py:609, in ColumnBase.fillna(self, value, method, dtype)
599 def fillna(
600 self: T,
601 value: Any = None,
602 method: str = None,
603 dtype: Dtype = None,
604 ) -> T:
605 """Fill null values with ``value``.
606
607 Returns a copy with null filled.
608 """
--> 609 return libcudf.replace.replace_nulls(
610 input_col=self, replacement=value, method=method, dtype=dtype
611 )

File replace.pyx:139, in cudf._lib.replace.replace_nulls()

File /datasets/vjawa/miniconda3/envs/cugraph_dgl_dev/lib/python3.9/site-packages/cudf/core/single_column_frame.py:95, in SingleColumnFrame.__bool__(self)
94 def __bool__(self):
---> 95 raise TypeError(
96 f"The truth value of a {type(self)} is ambiguous. Use "
97 "a.empty, a.bool(), a.item(), a.any() or a.all()."
98 )

TypeError: The truth value of a is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all().
```

**Expected behavior**

I expect this to work like it does for non list columns.

```python3
import cudf
s1 = cudf.Series(['game', 'game', None,None,None])
s2 = cudf.Series([1, 1, None,None,None])
df = cudf.DataFrame({'_TYPE_':s1, 'h':s2})

c1 = cudf.Series([None,None, 'user', 'user', 'user'])
c2 = cudf.Series([None, None, 2, 2, 2])
subdf = cudf.DataFrame({'_TYPE_':c1, 'h':c2})
df.fillna(subdf)
```
```python3
_TYPE_ h
0 game 1
1 game 1
2 user 2
3 user 2
4 user 2
```

**Environment overview (please complete the following information)**
- Method of cuDF install: [conda]
cudf 22.12.00a221128 cuda_11_py39_g0c60819cf3_305 rapidsai-nightly

**Additional context**

Impacts property graph in cugraph where we use it for replacing fillna

https://github.com/rapidsai/cugraph/blob/cb0d0923616f656ec816f999aa633ecbf3c57267/python/cugraph/cugraph/structure/property_graph.py#L758

https://github.com/rapidsai/cugraph/blob/cb0d0923616f656ec816f999aa633ecbf3c57267/python/cugraph/cugraph/structure/property_graph.py#L1155

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.