MLBazaar / MLBazaar/MLPrimitives

Primitive for normalizing (feature scaling) input data

Open
#82 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

new primitives Pending Review
Dominant language
Python
Stars
70
Forks
37
PR merge metrics
No merged PRs in 30d

Description

I want to create a primitive for normalization of data, or feature scaling so that the input is rescaled to [-1,1]

The formula for rescaling is
X_rescaled = (X - X.min) / (X.max - X.min)

The primitive input arguments are:

  • data (pandas dataframe)
  • column (string): the column to be rescaled
  • trim_percentage (float): percentage from the bottom and top to trim
  • inplace=True: if false, creates a new column called rescaled_input

Here are some potential issues with the implementation:

  • the implementation will run on historic data. If this primitive were to be used in an online system, we would have to either implement dynamic rescaling of dataset or automatically flag values larger than min and max as anomalies. Or we can just clip the input and assign large values as -1 or 1. The last suggestion makes the most sense, in my opinion.

  • distribution of the data. The data may have an outlier (very large value, e.g. 1234) and then the remaining values would be somewhere between -10 and 10. This would result in bad rescaling. One solution is to trim the lowest and highest 1% values, but what if those were outliers?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing existing Python primitive conventions and how pandas DataFrames are handled in this repository. Resolve whether the target range is [-1,1] or the supplied formula's [0,1], along with the intended trimming, clipping, and inplace behavior. Done means the normalization primitive and its edge cases are covered by tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas, python
Domain
data, machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.