[FEA] support numeric_only for DataFrame.corr
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
**Is your feature request related to a problem? Please describe.**
working with `import cudf as pd`
**Describe the solution you'd like**
`cudf.DataFrame.corr` matching `pandas.DataFrame.corr` behavior
1. addition of `numeric_only` parameter
2. default `numeric_only=None` with deprecation warning and lifecycle similar to `pandas`
```
In [1]: import cudf as pd
In [2]: pd.__version__
Out[2]: '22.12.0'
In [3]: df = pd.DataFrame({'a': range(10), 'b': range(10,20), 'c': list('zyxwvutsrq')})
In [4]: df.corr()
---------------------------------------------------------------------------
KeyError Traceback (most recent call last)
Cell In [4], line 1
----> 1 df.corr()
File ~/.local/lib/python3.9/site-packages/cudf/core/dataframe.py:6490, in DataFrame.corr(self, method, min_periods)
6470 """Compute the correlation matrix of a DataFrame.
6471
6472 Parameters
(...)
6487 The requested correlation matrix.
6488 """
6489 if method == "pearson":
-> 6490 values = self.values
6491 elif method == "spearman":
6492 values = self.rank().values
File ~/.local/lib/python3.9/site-packages/cudf/core/frame.py:433, in Frame.values(self)
420 @property
421 def values(self):
422 """
423 Return a CuPy representation of the DataFrame.
424
(...)
431 The values of the DataFrame.
432 """
--> 433 return self.to_cupy()
File ~/.local/lib/python3.9/site-packages/nvtx/nvtx.py:101, in annotate.__call__..inner(*args, **kwargs)
98 @wraps(func)
99 def inner(*args, **kwargs):
100 libnvtx_push_range(self.attributes, self.domain.handle)
--> 101 result = func(*args, **kwargs)
102 libnvtx_pop_range(self.domain.handle)
103 return result
File ~/.local/lib/python3.9/site-packages/cudf/core/frame.py:533, in Frame.to_cupy(self, dtype, copy, na_value)
507 @_cudf_nvtx_annotate
508 def to_cupy(
509 self,
(...)
512 na_value=None,
513 ) -> cupy.ndarray:
514 """Convert the Frame to a CuPy array.
515
516 Parameters
(...)
531 cupy.ndarray
532 """
--> 533 return self._to_array(
534 (lambda col: col.values.copy())
535 if copy
536 else (lambda col: col.values),
537 cupy.empty,
538 dtype,
539 na_value,
540 )
File ~/.local/lib/python3.9/site-packages/cudf/core/frame.py:498, in Frame._to_array(self, get_column_values, make_empty_matrix, dtype, na_value)
491 matrix = make_empty_matrix(
492 shape=(len(self), ncol), dtype=dtype, order="F"
493 )
494 for i, col in enumerate(self._data.values()):
495 # TODO: col.values may fail if there is nullable data or an
496 # unsupported dtype. We may want to catch and provide a more
497 # suitable error.
--> 498 matrix[:, i] = get_column_values_na(col)
499 return matrix
File cupy/_core/core.pyx:1508, in cupy._core.core.ndarray.__setitem__()
File cupy/_core/_routines_indexing.pyx:51, in cupy._core._routines_indexing._ndarray_setitem()
File cupy/_core/_routines_indexing.pyx:997, in cupy._core._routines_indexing._scatter_op()
File cupy/_core/_kernel.pyx:1292, in cupy._core._kernel.ufunc.__call__()
File cupy/_core/_kernel.pyx:1319, in cupy._core._kernel.ufunc._get_ufunc_kernel()
File cupy/_core/_kernel.pyx:1025, in cupy._core._kernel._get_ufunc_kernel()
File cupy/_core/_kernel.pyx:66, in cupy._core._kernel._get_simple_elementwise_kernel()
File cupy/_core/_kernel.pyx:322, in cupy._core._kernel._get_kernel_params()
File cupy/_core/_kernel.pyx:298, in cupy._core._kernel._ArgInfo.get_param_c_type()
File cupy/_core/_kernel.pyx:285, in cupy._core._kernel._ArgInfo.get_c_type()
File cupy/_core/_scalar.pyx:68, in cupy._core._scalar.get_typename()
File cupy/_core/_scalar.pyx:73, in cupy._core._scalar.get_typename()
KeyError:
In [5]: df.to_pandas().corr()
:1: FutureWarning: The default value of numeric_only in DataFrame.corr is deprecated. In a future version, it will default to False. Select only valid columns or specify the value of numeric_only to silence this warning.
df.to_pandas().corr()
Out[5]:
a b
a 1.0 1.0
b 1.0 1.0
```
[`pandas` introduced a `numeric_only` parameter](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.corr.html)
```
In [6]: df.to_pandas().corr(numeric_only=True)
Out[6]:
a b
a 1.0 1.0
b 1.0 1.0
In [7]: df.corr(numeric_only=True)
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
Cell In [7], line 1
----> 1 df.corr(numeric_only=True)
TypeError: corr() got an unexpected keyword argument 'numeric_only'
```
```
Contributor guide
Assessment
This issue has not been assessed yet.