[BUG] fillna with dataframe containing list dtypes fails
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
**Describe the bug**
fillna with dataframe containing list dtypes fails .
**Steps/Code to reproduce bug**
```python3
import cudf
s1 = cudf.Series(['game', 'game', None,None,None])
s2 = cudf.Series([[1], [1], None,None,None])
df = cudf.DataFrame({'_TYPE_':s1, 'h':s2})
c1 = cudf.Series([None,None, 'user', 'user', 'user'])
c2 = cudf.Series([None, None, [2], [2], [2]])
subdf = cudf.DataFrame({'_TYPE_':c1, 'h':c2})
df.fillna(subdf)
```
```python3
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
Cell In [23], line 9
7 c2 = cudf.Series([None, None, [2], [2], [2]])
8 subdf = cudf.DataFrame({'_TYPE_':c1, 'h':c2})
----> 9 df.fillna(subdf)
File /datasets/vjawa/miniconda3/envs/cugraph_dgl_dev/lib/python3.9/contextlib.py:79, in ContextDecorator.__call__..inner(*args, **kwds)
76 @wraps(func)
77 def inner(*args, **kwds):
78 with self._recreate_cm():
---> 79 return func(*args, **kwds)
File /datasets/vjawa/miniconda3/envs/cugraph_dgl_dev/lib/python3.9/site-packages/cudf/core/indexed_frame.py:1842, in IndexedFrame.fillna(self, value, method, axis, inplace, limit)
1837 @_cudf_nvtx_annotate
1838 def fillna(
1839 self, value=None, method=None, axis=None, inplace=False, limit=None
1840 ): # noqa: D102
1841 old_index = self._index
-> 1842 ret = super().fillna(value, method, axis, inplace, limit)
1843 if inplace:
1844 self._index = old_index
File /datasets/vjawa/miniconda3/envs/cugraph_dgl_dev/lib/python3.9/contextlib.py:79, in ContextDecorator.__call__..inner(*args, **kwds)
76 @wraps(func)
77 def inner(*args, **kwds):
78 with self._recreate_cm():
---> 79 return func(*args, **kwds)
File /datasets/vjawa/miniconda3/envs/cugraph_dgl_dev/lib/python3.9/site-packages/cudf/core/frame.py:905, in Frame.fillna(self, value, method, axis, inplace, limit)
899 should_fill = (
900 col_name in value
901 and col.contains_na_entries
902 and not libcudf.scalar._is_null_host_scalar(replace_val)
903 ) or method is not None
904 if should_fill:
--> 905 filled_data[col_name] = col.fillna(replace_val, method)
906 else:
907 filled_data[col_name] = col.copy(deep=True)
File /datasets/vjawa/miniconda3/envs/cugraph_dgl_dev/lib/python3.9/site-packages/cudf/core/column/column.py:609, in ColumnBase.fillna(self, value, method, dtype)
599 def fillna(
600 self: T,
601 value: Any = None,
602 method: str = None,
603 dtype: Dtype = None,
604 ) -> T:
605 """Fill null values with ``value``.
606
607 Returns a copy with null filled.
608 """
--> 609 return libcudf.replace.replace_nulls(
610 input_col=self, replacement=value, method=method, dtype=dtype
611 )
File replace.pyx:139, in cudf._lib.replace.replace_nulls()
File /datasets/vjawa/miniconda3/envs/cugraph_dgl_dev/lib/python3.9/site-packages/cudf/core/single_column_frame.py:95, in SingleColumnFrame.__bool__(self)
94 def __bool__(self):
---> 95 raise TypeError(
96 f"The truth value of a {type(self)} is ambiguous. Use "
97 "a.empty, a.bool(), a.item(), a.any() or a.all()."
98 )
TypeError: The truth value of a is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all().
```
**Expected behavior**
I expect this to work like it does for non list columns.
```python3
import cudf
s1 = cudf.Series(['game', 'game', None,None,None])
s2 = cudf.Series([1, 1, None,None,None])
df = cudf.DataFrame({'_TYPE_':s1, 'h':s2})
c1 = cudf.Series([None,None, 'user', 'user', 'user'])
c2 = cudf.Series([None, None, 2, 2, 2])
subdf = cudf.DataFrame({'_TYPE_':c1, 'h':c2})
df.fillna(subdf)
```
```python3
_TYPE_ h
0 game 1
1 game 1
2 user 2
3 user 2
4 user 2
```
**Environment overview (please complete the following information)**
- Method of cuDF install: [conda]
cudf 22.12.00a221128 cuda_11_py39_g0c60819cf3_305 rapidsai-nightly
**Additional context**
Impacts property graph in cugraph where we use it for replacing fillna
https://github.com/rapidsai/cugraph/blob/cb0d0923616f656ec816f999aa633ecbf3c57267/python/cugraph/cugraph/structure/property_graph.py#L758
https://github.com/rapidsai/cugraph/blob/cb0d0923616f656ec816f999aa633ecbf3c57267/python/cugraph/cugraph/structure/property_graph.py#L1155
Contributor guide
Assessment
This issue has not been assessed yet.