NVIDIA / NVIDIA/cudf

`describe` casts all values to `str` for several types

Open
#9,900 3 comments 0 reactions 0 assignees View on GitHub
bug Python
Dominant language
C++
Stars
9.8k
Forks
1.1k
Avg merge
3d 6m
Merged PRs (30d)
278

Description

The [`describe` implementations](https://github.com/rapidsai/cudf/blob/b3b299ae22f31adb8f380d7add1ce2bdb726ab26/python/cudf/cudf/core/series.py#L3317-L3423), especially for datetime and timedelta types, appear to be casting all the values to `str` (aside from the numeric implementation). This does not align with Pandas behavior:
```python
>>> s = pd.Series([
... np.datetime64("2000-01-01"),
... np.datetime64("2010-01-01"),
... np.datetime64("2010-01-01"),
... ])
>>> print(type(s.describe()["top"]))

```

I recognize there is an issue here with different types, namely that `count` and `freq` are not of the same type as `mean`, `min`, percentiles, or `max`. This also affects numerical columns which will upcast integer values like `count` to floating types.

Some options to resolve this (and their downsides):
1. Current implementation: return all values as `str` (results are on GPU ...but data is not usable as `str` type).
2. Return a `pd.DataFrame` or `dict` that can have multiple types (not a GPU DataFrame).

I propose changing behavior to adopt option (2), and return a `pd.DataFrame`. The summary doesn't really need to be a GPU DataFrame since it contains so few values. (Do we have precedent for this kind of behavior returning a CPU (Pandas) DataFrame?)

_Originally posted by @bdice in https://github.com/rapidsai/cudf/pull/9867#discussion_r768149355_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.