INRIA / INRIA/scikit-learn-mooc

Add warning messages when full data-set is used

Open
#441 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
1.4k
Forks
600
Avg merge
6d 20h
Merged PRs (30d)
2

Description

The full data-set (no train-test split or cv) is used for modeling in the following notebooks:

- [linear_regression_without_sklearn.py](https://inria.github.io/scikit-learn-mooc/python_scripts/linear_regression_without_sklearn.html)
- [linear_models_ex_01.py](https://inria.github.io/scikit-learn-mooc/python_scripts/linear_models_ex_01.html) and its [solution](https://inria.github.io/scikit-learn-mooc/python_scripts/linear_models_sol_01.html)
- [linear_regression_in_sklearn.py](https://inria.github.io/scikit-learn-mooc/python_scripts/linear_regression_in_sklearn.html)
- [linear_models_ex_02.py](https://inria.github.io/scikit-learn-mooc/python_scripts/linear_models_ex_01.html) and its [solution](https://inria.github.io/scikit-learn-mooc/python_scripts/linear_models_sol_02.html)
- [linear_regression_non_linear_link.py](https://inria.github.io/scikit-learn-mooc/python_scripts/linear_regression_non_linear_link.html)
- [linear_models_ex_04.py](https://inria.github.io/scikit-learn-mooc/python_scripts/linear_models_ex_01.html) and its [solution](https://inria.github.io/scikit-learn-mooc/python_scripts/linear_models_sol_04.html)
- [logistic_regression_non_linear.py](https://inria.github.io/scikit-learn-mooc/python_scripts/logistic_regression_non_linear.html)
- [trees_regression.py](https://inria.github.io/scikit-learn-mooc/python_scripts/trees_regression.html)
- [trees_ex_02.py](https://inria.github.io/scikit-learn-mooc/python_scripts/trees_ex_01.html) and its [solution](https://inria.github.io/scikit-learn-mooc/python_scripts/trees_sol_02.html)
- [ensemble_bagging.py](https://inria.github.io/scikit-learn-mooc/python_scripts/ensemble_bagging.html)
- [ensemble_adaboost.py](https://inria.github.io/scikit-learn-mooc/python_scripts/ensemble_adaboost.html)

This has been a source of confusion (see for instance [this forum question](https://mooc-forums.inria.fr/moocsl/t/split-train-and-test/10658)).

We should add a _Warning_ message similar (but adapted to each case) to the one in [logistic_regression_non_linear.py](https://inria.github.io/scikit-learn-mooc/python_scripts/logistic_regression_non_linear.html):

> Warning: Be aware that we fit and will check the boundary decision of the classifier on the same dataset without splitting the dataset into a training set and a testing set. While this is a bad practice, we use it for the sake of simplicity to depict the model behavior. Always use cross-validation when you want to assess the generalization performance of a machine-learning model.

Additionally, a _Warning_ message should be added in the following notebooks

- [linear_models_ex_01.py](https://inria.github.io/scikit-learn-mooc/python_scripts/linear_models_ex_01.html) and its [solution](https://inria.github.io/scikit-learn-mooc/python_scripts/linear_models_sol_01.html)
- [linear_regression_in_sklearn.py](https://inria.github.io/scikit-learn-mooc/python_scripts/linear_regression_in_sklearn.html)
- [linear_models_ex_02.py](https://inria.github.io/scikit-learn-mooc/python_scripts/linear_models_ex_01.html) and its [solution](https://inria.github.io/scikit-learn-mooc/python_scripts/linear_models_sol_02.html)
- [linear_regression_non_linear_link.py](https://inria.github.io/scikit-learn-mooc/python_scripts/linear_regression_non_linear_link.html)

where we remind the user that scoring the model in the full data-set is not necessarily wrong but provides no info about under/over-fitting.

What do you think?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.