dask / dask/dask-ml

Candidate transformers

Open
#6 13 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
951
Forks
262
PR merge metrics
No merged PRs in 30d

Description

I think it'd be nice to have some transformers that work on dask and numpy arrays, & dask and pandas DataFrames. This would be good since

1. We can depend on dask and pandas, scikit-learn can't
2. A basic transformer is generally much less work than a full-blown estimato

## API

See https://github.com/tomaugspurger/sktransformers for some inspiration (and maybe some tests)?

- We should match scikit-learn as closely as possible where things overlap
- All transformers should take an optional `columns` argument. When specified, the transformation will be limited to just `columns` (e.g. if doing a standard scaling, and `columns=['A', 'B']`, only `'A'` and `'B'` are scaled). By default, all columns are scaled
- We should operate on `np.ndarray`, `dask.array.Array`, `pandas.core.NDFrame`, `dask.dataframe._Frame`.
- Should our operations be nan-safe?

The big question right now is should fitting be eager, and fitted values concrete? e.g.

```python
scaler = StandardScaler()
scaler.fit(X) # X is a dask.array
```

So, has `scaler.mean_` been computed, and is it a `dask.array` or a `numpy.array`? This is a big decision.

Candidates

- [ ] [Imputer](http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.Imputer.html)
- [ ] [CategoricalEncoder](https://github.com/TomAugspurger/sktransformers/blob/992fddfaab0cfc9fea83e68e4a345f7f2de11990/sktransformers/preprocessing.py#L41) (TODO: check on Joris' recent work in sklearn here)
- [ ] [DummyEncoder](https://github.com/TomAugspurger/sktransformers/blob/992fddfaab0cfc9fea83e68e4a345f7f2de11990/sktransformers/preprocessing.py#L77)
- [ ] [VarianceThreshold](http://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.VarianceThreshold.html)
- [x] [MinMaxScaler](http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.MinMaxScaler.html)
- [ ] [PolynomialFeatures](http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.PolynomialFeatures.html)
- [x] [QuantileTransformer](http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.QuantileTransformer.html)
- [ ] [High-Cardinality Categorical](http://hccencoding-project.readthedocs.io/en/latest/) (see https://gist.github.com/ogrisel/b6a97ed87939e3b559568ac2f6599cba)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.