Pandas categorical optimization
- Dominant language
- Python
- Stars
- 1k
- Forks
- 131
- PR merge metrics
- No merged PRs in 30d
Description
Datashader issue https://github.com/bokeh/datashader/issues/396 revealed a bottleneck in `odo`'s pandas backend [odo/backends/pandas.py]() on this line: https://github.com/blaze/odo/blob/master/odo/backends/pandas.py#L22, which occurs whenever the input column is of type `'category'` . Here is my proposed fix:
```
def dshape_from_pandas(col):
if isinstance(col.dtype, categorical):
cat_dshape = dshape('{} * {}'.format(
len(col.cat.categories),
col.cat.categories.dtype,
))
return Categorical(
col.cat.categories.values,
type=cat_dshape,
ordered=col.cat.categorical.ordered,
)
...
```
There is potential for an additional bottleneck to occur inside `datashape`'s `Categorical` constructor here: https://github.com/blaze/datashape/blob/master/datashape/coretypes.py#L1106, where the first argument is coerced into a `tuple`. At first glance it seems straightforward to simply drop the type coercion, but a symmetrical refactor to the `dshape()` functionality is necessary to get `datashape`'s unit tests to pass.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.