EpistasisLab / EpistasisLab/tpot
Use of standardScaler() and TPOT
- Dominant language
- Jupyter Notebook
- Stars
- 10.1k
- Forks
- 1.6k
- PR merge metrics
- No merged PRs in 30d
Description
It is good practice to first split the data into train and test, then scale by `fit_transform` on the training set, and `transform` on the test set.
So I create a train test split:
```python
training_features, testing_features, training_target, testing_target = \
train_test_split(data, labels, random_state=1,stratify=labels.values,test_size = 0.2)
```
Scale the data
```python
sc = StandardScaler()
training_features_scaled = sc.fit_transform(training_features)
testing_features_scaled = sc.transform(testing_features)
```
Now I feed `training_features_scaled` into `TPOT`.
However, TPOT then runs cross fold validation on this dataset, which bascially means that in this case, scaling has been done before the split and there's a potential for leakage between train and test during each fold. Is this something that we should avoid, and if so, how can we avoid this?
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.