GoogleCloudPlatform / GoogleCloudPlatform/training-data-analyst

Need to add data shuffling operation in building_production_ml_system labs

Open
#1,376 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
8.6k
Forks
6.1k
Avg merge
4h 44m
Merged PRs (30d)
2

Description

## Issue
Since BQ automatically sort data, data creation query should be shuffled explicitly to avoid unintentional sorting.
Without it, the sorted data will cause overfit problem during model training.

https://github.com/GoogleCloudPlatform/training-data-analyst/blob/f4d390f47681980b201ddc71bea30891c0812975/courses/machine_learning/deepdive2/building_production_ml_systems/solutions/0_export_data_from_bq_to_gcs.ipynb#L117-L144

## Solution
One of the quick workarounds is to add `ORDER BY RAND()` at the last line of SQL.

## Evidence
Without shuffling, created data is sorted unintentionally.

```python
import pandas as pd
import matplotlib.pyplot as plt
cols = ['fare_amount', 'datetime', 'pickuplon', 'pickuplat', 'dropofflon', 'dropofflat','passengers', 'key']
df = pd.read_csv('taxi-train-000000000000.csv', names=cols)

for c in df.columns:
try:
df[c].plot(title=c)
plt.show()
except:
pass
```

![image](https://user-images.githubusercontent.com/6895245/105989756-ea9e8780-60e4-11eb-908b-6cc50ca1c9a4.png)
![image](https://user-images.githubusercontent.com/6895245/105989947-246f8e00-60e5-11eb-9029-4f3f51e6ef3a.png)
![image](https://user-images.githubusercontent.com/6895245/105989988-2e918c80-60e5-11eb-8964-f9b3ba08a27f.png)
![image](https://user-images.githubusercontent.com/6895245/105990004-34876d80-60e5-11eb-83ec-5165253ac89e.png)
![image](https://user-images.githubusercontent.com/6895245/105990027-3d783f00-60e5-11eb-8cc3-258f9981982e.png)

![image](https://user-images.githubusercontent.com/6895245/105989816-fc802a80-60e4-11eb-9c0c-5bfaa8cc4add.png)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.