NVIDIA / NVIDIA/cudf

[FEA]Coalesce(), find the first non-null value, no equivalent function in RAPIDS

Open
#6,759 4 comments 0 reactions 0 assignees View on GitHub
0 - Backlog dask feature request libcudf Python
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

I want to create a new column, which is the first non-null value of several columns, I used the function [coalesce()](https://www.rdocumentation.org/packages/dplyr/versions/0.7.8/topics/coalesce) in dplyr to achieve that by simply run ` data %>% mutate(d = coalesce(a, b, c))`, in Dask CPU, I run below code to achieve that
```
d = {'a': [1, 2, 3, None, 5, 1, None, None, None, None, 3, None, 5, 6, 7], \
'b': [1, 2, 3, 200, 5, 1, 200, None, 400, None, 3, None, 5, 6, 7], \
'c': [1, 2, 3, None, 5, 1, 300, 300, None, 600, 3, 300, 5, 6, 7]}

df2 = pd.DataFrame(d)

ddf2 = dd.from_pandas(df2,npartitions=50)
ddf2['d'] = ddf2['a'].copy().fillna(ddf2['b']).fillna(ddf2['c'])
ddf2.compute()
```
However, in dask cuDF, when I am trying to use the fillna() to acheive my intent, I got the error message
```
d = {'id': ['a', 'a','a','a','a','b','b','b','b','b','c','c','c','c','c'], \
'time': ['1', '2', '4', '3', '5', '1', '2', '3', '4', '5','1', '2', '3', '4', '5'], \
'a': [1, 2, 3, None, 5, 1, None, None, None, None, 3, None, 5, 6, 7], \
'b': [1, 2, 3, 200, 5, 1, 200, None, 400, None, 3, None, 5, 6, 7], \
'c': [1, 2, 3, None, 5, 1, 300, 300, None, 600, 3, 300, 5, 6, 7]}

df = pd.DataFrame(data=d)
gdf = cudf.DataFrame.from_pandas(df)

ddf = dask_cudf.from_cudf(gdf, npartitions=4)

ddf['d'] = ddf['a'].copy().fillna(ddf['b']).fillna(ddf['c'])
ddf.compute()
```
error message: `RuntimeError: cuDF failure at: /opt/conda/envs/rapids/conda-bld/libcudf_1598487636199/work/cpp/src/replace/replace.cu:804: Column size mismatch`

So I am wondering if there are functions that allows me to get the first non-null value from several columns in cuDF or if we can add that.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.