dotnet / dotnet/machinelearning

Rolling Cross-validation for Time-series

Open
#1,026 1 comment 8 reactions 0 assignees View on GitHub
area-TimeSeries enhancement Priority:2
Dominant language
C#
Stars
9.4k
Forks
2k
Avg merge
2d 20h
Merged PRs (30d)
11

Description

To properly handle time-series (and time-dependent data in general), we should implement a Rolling Cross-validation to add to our existing CV & TrainTest modes.

We are currently merging various time-series functionality from the internal repo to this repo via https://github.com/dotnet/machinelearning/pull/977 _"Port Time Series"_. This PR does not include a rolling cross-validation, used heavily in time-series tasks.

Rolling CV is better for time dependent datasets by always testing on data which is _newer_ than the training data. Standard CV leaks future data in to the training set. Other names of Rolling CV include { walk-forward / roll-forward / rolling origin / window } CV.

Background on method:
http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.TimeSeriesSplit.html
https://otexts.org/fpp2/accuracy.html#time-series-cross-validation
https://stats.stackexchange.com/questions/14099/using-k-fold-cross-validation-for-time-series-model-selection
https://robjhyndman.com/hyndsight/tscv/
https://www.kaggle.com/c/recruit-restaurant-visitor-forecasting/discussion/46602
https://towardsdatascience.com/time-series-nested-cross-validation-76adba623eb9

To further investigate missing time-series components, the [Azure ML Forecasting Toolkit](https://docs.microsoft.com/en-us/python/api/azuremlftk/) is a good package listing components needed for this task:
* Metrics: [MAPE, MASE_single_grain, SMAPE](https://docs.microsoft.com/en-us/python/api/azuremlftk/ftk.metrics.metrics?view=azure-forecasting-py)
* Models: [ARIMA, etc](https://docs.microsoft.com/en-us/python/api/azuremlftk/ftk.models?view=azure-forecasting-py)
* etc

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.