apache / apache/datafusion

It will be good to have Bucketizer and OneHotEncoder in DataFusion like PySpark

Open
#4,195 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

Feature engineering before Machine Learning needs some special transformation to deal with Categorical Data either they are nominal or Ordinal. It will be nice to have built-in OneHot encoder and Bucketizer functions. Currently, it is possible to create such encoded values but that needs a lot of boilerplate code with Joins and When else statements.

For Bucketizer up on providing a vector/list of range it will be able to create a new column in the data frame which will bucketize the input continuous column. An example can be found in [PySpark API doc](https://spark.apache.org/docs/latest/api/python/reference/api/pyspark.ml.feature.Bucketizer.html)
For OneHotEncoder upon providing a column if the column has n category it will be able to create `n` or `n-1` column up on giving a True False parameter. Here is [PySpark API doc](https://spark.apache.org/docs/3.1.1/api/python/reference/api/pyspark.ml.feature.OneHotEncoder.html).

Any Alternative ideal also should work. The new features can be under a new module `datafusion::ml::Bucketizer` and `datafusion::ml::OneHotEncoder`

Contributor guide

Open the contributing guide

Research direction

Start with the proposed datafusion::ml::Bucketizer and datafusion::ml::OneHotEncoder module, then compare the linked PySpark Bucketizer and OneHotEncoder API documentation. Define the supported range-list and category-column behavior, including the n versus n-1 option; done means both transformations are available without the described join and when/else boilerplate.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, rust
Domain
data-engineering, machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.