scikit-learn / scikit-learn/scikit-learn

[Feature Request] RollingTimeSeriesSplit: Fixed-window cross-validation with right alignment

Open
#33,520 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Needs Decision - Include Feature New Feature
Dominant language
Python
Stars
67.3k
Forks
27.4k
Avg merge
1d 15h
Merged PRs (30d)
58

Description

[!WARNING]
This issue is not yet ready for a PR. If you are interested in contributing to scikit-learn, please have a look at our contributing guidelines, and in particular the sections for new contributors and the "Needs triage" label.

Describe the workflow you want to enable

In many real-world time series forecasting scenarios (such as financial modeling, demand forecasting, or predictive maintenance), data undergoes concept drift. Training a model on very old data can actually hurt its performance on recent data.

Because of this, the standard Expanding Window cross-validation (where the training set grows indefinitely, which is the default behavior of TimeSeriesSplit) is often inappropriate. Instead, practitioners need a Fixed-Size Sliding Window (also known as Walk-Forward Validation).

Furthermore, in continuous retraining workflows, we only care about evaluating the model on the most recent data blocks. If a dataset length is not perfectly divisible by the window size, the standard approach either drops the most recent data or forces overlapping.

The workflow I want to enable allows users to easily do the following:

  1. Define a fixed train_size (e.g., train on exactly 30 days).
  2. Define a fixed test_size (e.g., test on exactly 7 days).
  3. Define a specific step_size (e.g., slide the window by 7 days for disjoint sets, or 1 day for rolling origin).
  4. Ensure the splits are right-aligned (anchored to the end of the dataset), so the final test set evaluates on the absolute latest available data, shifting any historical "remainder" to the oldest training set.

Currently, achieving this with TimeSeriesSplit requires fragile, manual calculations of max_train_size and n_splits based on len(X), which is highly impractical in dynamic data pipelines where len(X) changes daily.

Describe your proposed solution

I propose adding a new cross-validator class named RollingTimeSeriesSplit to sklearn.model_selection.

Unlike TimeSeriesSplit which is built around the n_splits parameter, RollingTimeSeriesSplit is built around absolute window sizes (train_size and test_size).

Proposed API:

python

cv = RollingTimeSeriesSplit(
    train_size=100,  # Fixed size for the training window
    test_size=20,    # Fixed size for the testing window
    step_size=20,    # How far the window slides (default: train_size)
    gap=0            # Optional embargo between train and test
)

Key Implementation Details:

  1. Dynamic n_splits: The number of splits is no longer a required input. It is calculated dynamically via get_n_splits(X) based on the total length of the dataset and the window parameters.
  2. Right-Alignment Logic: The algorithm calculates the splits starting from the end of the array (n_samples). This guarantees that the very last test set always perfectly covers the most recent data.
  3. Remainder Handling: If the total number of samples is not perfectly divisible by the block size/step, the "remainder" (the oldest historical data points) is safely absorbed into the first training set.

By introducing this as a separate class rather than overloading TimeSeriesSplit, we maintain clear separation of concerns (Expanding Window vs. Fixed Sliding Window) without breaking backward compatibility or complicating the existing API.

Describe alternatives you've considered, if relevant

I considered two main alternatives before proposing a new RollingTimeSeriesSplit class:

Alternative 1: Modifying the existing TimeSeriesSplit
I initially thought about adding train_size, test_size, and step_size parameters to the existing TimeSeriesSplit. However, TimeSeriesSplit fundamentally relies on n_splits to calculate variable window sizes dynamically. Mixing an "expanding window" logic (based on n_splits) with a "fixed sliding window" logic (ignoring n_splits and computing it from X) within the same class makes the API confusing and heavily pollutes the __init__ and _split methods with conflicting parameters. A separate class ensures a much cleaner and intuitive API.

Alternative 2: Forcing TimeSeriesSplit to act like a rolling window manually
Currently, users try to hack the existing class by strictly setting max_train_size and calculating n_splits manually based on len(X).
Example: n_splits = (len(X) - train_size) // test_size.
However, this workaround fails in dynamic pipelines where the length of incoming data changes daily. It also defaults to "left-alignment" (starting from index 0), meaning if there is a remainder, the most recent data points are excluded from the final test set, which is unacceptable for time-series forecasting.

Additional context

To add some context, time series data often suffers from non-stationarity and concept drift (e.g., financial markets changing regimes, or post-COVID retail demand). In these scenarios, training a model on data from 5 years ago often introduces noise rather than useful signal.

Therefore, accumulating data indefinitely (the current TimeSeriesSplit behavior) degrades model performance over time. The industry standard for these specific time series is Walk-Forward Optimization using a strict rolling window.

Visually, this is the difference between the two approaches:

Current TimeSeriesSplit (Expanding Window):

| Train | Test |
| Train ....... | Test |
| Train ............... | Test |

(Old data is never forgotten)

Proposed RollingTimeSeriesSplit (Sliding Window):

| Train | Test |
        | Train | Test |
                | Train | Test |

(Old data is discarded, maintaining a fixed-size lookback window)

Adding this class brings scikit-learn's cross-validation capabilities closer to specialized time-series libraries, addressing a very frequent pain point for data scientists working with non-stationary data.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the contributing guidelines and the existing TimeSeriesSplit entry point in sklearn.model_selection. Review how fixed train and test sizes, step_size, gap, dynamic split counts, and right alignment would fit the existing cross-validation API. The issue is done only after the proposed API and behavior are agreed and the new cross-validator is implemented accordingly.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning, testing-qa
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.