MLBazaar / MLBazaar/MLPrimitives
Primitive for normalizing (feature scaling) input data
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 70
- Forks
- 37
- PR merge metrics
- No merged PRs in 30d
Description
I want to create a primitive for normalization of data, or feature scaling so that the input is rescaled to [-1,1]
The formula for rescaling is
X_rescaled = (X - X.min) / (X.max - X.min)
The primitive input arguments are:
data(pandas dataframe)column(string): the column to be rescaledtrim_percentage(float): percentage from the bottom and top to triminplace=True: if false, creates a new column calledrescaled_input
Here are some potential issues with the implementation:
-
the implementation will run on historic data. If this primitive were to be used in an online system, we would have to either implement dynamic rescaling of dataset or automatically flag values larger than
minandmaxas anomalies. Or we can just clip the input and assign large values as-1or1. The last suggestion makes the most sense, in my opinion. -
distribution of the data. The data may have an outlier (very large value,
e.g. 1234) and then the remaining values would be somewhere between-10 and 10. This would result in bad rescaling. One solution is to trim the lowest and highest 1% values, but what if those were outliers?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing existing Python primitive conventions and how pandas DataFrames are handled in this repository. Resolve whether the target range is [-1,1] or the supplied formula's [0,1], along with the intended trimming, clipping, and inplace behavior. Done means the normalization primitive and its edge cases are covered by tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- pandas, python
- Domain
- data, machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100