It will be good to have Bucketizer and OneHotEncoder in DataFusion like PySpark
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
Feature engineering before Machine Learning needs some special transformation to deal with Categorical Data either they are nominal or Ordinal. It will be nice to have built-in OneHot encoder and Bucketizer functions. Currently, it is possible to create such encoded values but that needs a lot of boilerplate code with Joins and When else statements.
For Bucketizer up on providing a vector/list of range it will be able to create a new column in the data frame which will bucketize the input continuous column. An example can be found in [PySpark API doc](https://spark.apache.org/docs/latest/api/python/reference/api/pyspark.ml.feature.Bucketizer.html)
For OneHotEncoder upon providing a column if the column has n category it will be able to create `n` or `n-1` column up on giving a True False parameter. Here is [PySpark API doc](https://spark.apache.org/docs/3.1.1/api/python/reference/api/pyspark.ml.feature.OneHotEncoder.html).
Any Alternative ideal also should work. The new features can be under a new module `datafusion::ml::Bucketizer` and `datafusion::ml::OneHotEncoder`
Contributor guide
Research direction
Start with the proposed datafusion::ml::Bucketizer and datafusion::ml::OneHotEncoder module, then compare the linked PySpark Bucketizer and OneHotEncoder API documentation. Define the supported range-list and category-column behavior, including the n versus n-1 option; done means both transformations are available without the described join and when/else boilerplate.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, rust
- Domain
- data-engineering, machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100