ageron / ageron/handson-ml3

[IDEA] Chapter 2, Improving the extra code for getting a uniform distribution for a feature.

Open
#103 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
14.1k
Forks
5.3k
PR merge metrics
No merged PRs in 30d

Description

Notebook name: 02_end_to_end_machine_learning_project
Section 3.3 “Feature Scaling”
Cell 95.

The code seems not robust as it assumes that consecutive percentiles have different values. This is not the case for e.g. the housing_median_age feature. Below is a suggested alternative.
```
# extra code – just shows that we get a uniform distribution
percentiles = pd.DataFrame(columns=['percentile'],
index=[p for p in range(1, 100)],
data=[np.percentile(housing["median_income"], p) for p in range(1, 100)])
percentiles.drop_duplicates(keep='last', inplace=True)
flattened_median_income = pd.cut(housing["median_income"],
bins=[-np.inf] + percentiles['percentile'].tolist() + [np.inf],
labels=percentiles.index.tolist() + [100])
flattened_median_income.hist(bins=len(percentiles) // 2 + 1)
plt.xlabel("Median income percentile")
plt.ylabel("Number of districts")
plt.show()
# Note: incomes below the 1st percentile are labeled 1, and incomes above the
# 99th percentile are labeled 100. This is why the distribution below ranges
# from 1 to 100 (not 0 to 100).
```

**Additional context**
Having said that, the housing_median_age indeed requires a different approach not only because of it’s multimodal distribution (as you state in the book) but also because there are multiple duplicate values in the distribution (probably as expected from an age feature that does not span across many years), which makes it difficult to break into evenly sized buckets.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.