GoogleCloudPlatform / GoogleCloudPlatform/training-data-analyst
Need to add data shuffling operation in building_production_ml_system labs
- Dominant language
- Jupyter Notebook
- Stars
- 8.6k
- Forks
- 6.1k
- Avg merge
- 4h 44m
- Merged PRs (30d)
- 2
Description
## Issue
Since BQ automatically sort data, data creation query should be shuffled explicitly to avoid unintentional sorting.
Without it, the sorted data will cause overfit problem during model training.
https://github.com/GoogleCloudPlatform/training-data-analyst/blob/f4d390f47681980b201ddc71bea30891c0812975/courses/machine_learning/deepdive2/building_production_ml_systems/solutions/0_export_data_from_bq_to_gcs.ipynb#L117-L144
## Solution
One of the quick workarounds is to add `ORDER BY RAND()` at the last line of SQL.
## Evidence
Without shuffling, created data is sorted unintentionally.
```python
import pandas as pd
import matplotlib.pyplot as plt
cols = ['fare_amount', 'datetime', 'pickuplon', 'pickuplat', 'dropofflon', 'dropofflat','passengers', 'key']
df = pd.read_csv('taxi-train-000000000000.csv', names=cols)
for c in df.columns:
try:
df[c].plot(title=c)
plt.show()
except:
pass
```






Contributor guide
Assessment
This issue has not been assessed yet.