facebook / facebook/prophet

Cross validation - initial parameter ≠ initial training period

Open
#2,330 5 comments 0 reactions 0 assignees View on GitHub
question
Dominant language
Python
Stars
20.4k
Forks
4.6k
Avg merge
19h 52m
Merged PRs (30d)
1

Description

Hello,

According to the [documentation](https://facebook.github.io/prophet/docs/diagnostics.html#cross-validation), the `initial` parameter should set the size of the initial training period.
> We specify the forecast horizon (horizon), and then optionally **the size of the initial training period (initial)** and the spacing between cutoff dates (period).

It seems that it's not working as intended. The models trained by cross validation are using all the data from the 1st point until the cutoff, `initial` has no impact on the training period.

I made some testing which confirms it:

```
import pandas as pd
from prophet import Prophet
from prophet.diagnostics import cross_validation
df = pd.read_csv('https://raw.githubusercontent.com/facebook/prophet/main/examples/example_wp_log_peyton_manning.csv')
df = df.head(1200)
m = Prophet()
m.fit(df)

# First try with initial = 730 days
df_cv1 = cross_validation(m, initial='730 days', period='180 days', horizon='90 days')
df_cv1.query("cutoff=='2010-02-15'")[['ds', 'yhat', 'cutoff']].head(3)
# ds yhat cutoff
# 2010-02-16 8.957788 2010-02-15
# 2010-02-17 8.724250 2010-02-15
# 2010-02-18 8.608022 2010-02-15

# Second try with initial = 400 days
df_cv2 = cross_validation(m, initial='400 days', period='180 days', horizon='90 days')
df_cv2.query("cutoff=='2010-02-15'")[['ds', 'yhat', 'cutoff']].head(5)
# ds yhat cutoff
# 2010-02-16 8.957788 2010-02-15
# 2010-02-17 8.724250 2010-02-15
# 2010-02-18 8.608022 2010-02-15

# Same results, even though we expected the training periods to be different
```

I've also checked the related source code, `initial` is not taken into account when defining training data :
https://github.com/facebook/prophet/blob/e665430adcd7690a1ea7565803f34043596045fe/python/prophet/diagnostics.py#L230-L239

Is this behaviour intended ?
Also wouldn't it make more sense that the folds in cross validation have a fixed size instead (rolling window of x days prior to the cutoff) ?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.