NVIDIA / NVIDIA/cudf

[FEA] support numeric_only for DataFrame.corr

Open
#12,626 0 comments 0 reactions 0 assignees View on GitHub
0 - Backlog feature request Python
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

**Is your feature request related to a problem? Please describe.**
working with `import cudf as pd`

**Describe the solution you'd like**
`cudf.DataFrame.corr` matching `pandas.DataFrame.corr` behavior

1. addition of `numeric_only` parameter
2. default `numeric_only=None` with deprecation warning and lifecycle similar to `pandas`

```
In [1]: import cudf as pd

In [2]: pd.__version__
Out[2]: '22.12.0'

In [3]: df = pd.DataFrame({'a': range(10), 'b': range(10,20), 'c': list('zyxwvutsrq')})

In [4]: df.corr()
---------------------------------------------------------------------------
KeyError Traceback (most recent call last)
Cell In [4], line 1
----> 1 df.corr()

File ~/.local/lib/python3.9/site-packages/cudf/core/dataframe.py:6490, in DataFrame.corr(self, method, min_periods)
6470 """Compute the correlation matrix of a DataFrame.
6471
6472 Parameters
(...)
6487 The requested correlation matrix.
6488 """
6489 if method == "pearson":
-> 6490 values = self.values
6491 elif method == "spearman":
6492 values = self.rank().values

File ~/.local/lib/python3.9/site-packages/cudf/core/frame.py:433, in Frame.values(self)
420 @property
421 def values(self):
422 """
423 Return a CuPy representation of the DataFrame.
424
(...)
431 The values of the DataFrame.
432 """
--> 433 return self.to_cupy()

File ~/.local/lib/python3.9/site-packages/nvtx/nvtx.py:101, in annotate.__call__..inner(*args, **kwargs)
98 @wraps(func)
99 def inner(*args, **kwargs):
100 libnvtx_push_range(self.attributes, self.domain.handle)
--> 101 result = func(*args, **kwargs)
102 libnvtx_pop_range(self.domain.handle)
103 return result

File ~/.local/lib/python3.9/site-packages/cudf/core/frame.py:533, in Frame.to_cupy(self, dtype, copy, na_value)
507 @_cudf_nvtx_annotate
508 def to_cupy(
509 self,
(...)
512 na_value=None,
513 ) -> cupy.ndarray:
514 """Convert the Frame to a CuPy array.
515
516 Parameters
(...)
531 cupy.ndarray
532 """
--> 533 return self._to_array(
534 (lambda col: col.values.copy())
535 if copy
536 else (lambda col: col.values),
537 cupy.empty,
538 dtype,
539 na_value,
540 )

File ~/.local/lib/python3.9/site-packages/cudf/core/frame.py:498, in Frame._to_array(self, get_column_values, make_empty_matrix, dtype, na_value)
491 matrix = make_empty_matrix(
492 shape=(len(self), ncol), dtype=dtype, order="F"
493 )
494 for i, col in enumerate(self._data.values()):
495 # TODO: col.values may fail if there is nullable data or an
496 # unsupported dtype. We may want to catch and provide a more
497 # suitable error.
--> 498 matrix[:, i] = get_column_values_na(col)
499 return matrix

File cupy/_core/core.pyx:1508, in cupy._core.core.ndarray.__setitem__()

File cupy/_core/_routines_indexing.pyx:51, in cupy._core._routines_indexing._ndarray_setitem()

File cupy/_core/_routines_indexing.pyx:997, in cupy._core._routines_indexing._scatter_op()

File cupy/_core/_kernel.pyx:1292, in cupy._core._kernel.ufunc.__call__()

File cupy/_core/_kernel.pyx:1319, in cupy._core._kernel.ufunc._get_ufunc_kernel()

File cupy/_core/_kernel.pyx:1025, in cupy._core._kernel._get_ufunc_kernel()

File cupy/_core/_kernel.pyx:66, in cupy._core._kernel._get_simple_elementwise_kernel()

File cupy/_core/_kernel.pyx:322, in cupy._core._kernel._get_kernel_params()

File cupy/_core/_kernel.pyx:298, in cupy._core._kernel._ArgInfo.get_param_c_type()

File cupy/_core/_kernel.pyx:285, in cupy._core._kernel._ArgInfo.get_c_type()

File cupy/_core/_scalar.pyx:68, in cupy._core._scalar.get_typename()

File cupy/_core/_scalar.pyx:73, in cupy._core._scalar.get_typename()

KeyError:

In [5]: df.to_pandas().corr()
:1: FutureWarning: The default value of numeric_only in DataFrame.corr is deprecated. In a future version, it will default to False. Select only valid columns or specify the value of numeric_only to silence this warning.
df.to_pandas().corr()
Out[5]:
a b
a 1.0 1.0
b 1.0 1.0
```

[`pandas` introduced a `numeric_only` parameter](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.corr.html)

```
In [6]: df.to_pandas().corr(numeric_only=True)
Out[6]:
a b
a 1.0 1.0
b 1.0 1.0

In [7]: df.corr(numeric_only=True)
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
Cell In [7], line 1
----> 1 df.corr(numeric_only=True)

TypeError: corr() got an unexpected keyword argument 'numeric_only'
```
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.