blaze / blaze/odo

Pandas categorical optimization

Open
#561 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1k
Forks
131
PR merge metrics
No merged PRs in 30d

Description

Datashader issue https://github.com/bokeh/datashader/issues/396 revealed a bottleneck in `odo`'s pandas backend [odo/backends/pandas.py]() on this line: https://github.com/blaze/odo/blob/master/odo/backends/pandas.py#L22, which occurs whenever the input column is of type `'category'` . Here is my proposed fix:

```
def dshape_from_pandas(col):
if isinstance(col.dtype, categorical):
cat_dshape = dshape('{} * {}'.format(
len(col.cat.categories),
col.cat.categories.dtype,
))
return Categorical(
col.cat.categories.values,
type=cat_dshape,
ordered=col.cat.categorical.ordered,
)
...
```

There is potential for an additional bottleneck to occur inside `datashape`'s `Categorical` constructor here: https://github.com/blaze/datashape/blob/master/datashape/coretypes.py#L1106, where the first argument is coerced into a `tuple`. At first glance it seems straightforward to simply drop the type coercion, but a symmetrical refactor to the `dshape()` functionality is necessary to get `datashape`'s unit tests to pass.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.