Cross validation - initial parameter ≠ initial training period
- Dominant language
- Python
- Stars
- 20.4k
- Forks
- 4.6k
- Avg merge
- 19h 52m
- Merged PRs (30d)
- 1
Description
Hello,
According to the [documentation](https://facebook.github.io/prophet/docs/diagnostics.html#cross-validation), the `initial` parameter should set the size of the initial training period.
> We specify the forecast horizon (horizon), and then optionally **the size of the initial training period (initial)** and the spacing between cutoff dates (period).
It seems that it's not working as intended. The models trained by cross validation are using all the data from the 1st point until the cutoff, `initial` has no impact on the training period.
I made some testing which confirms it:
```
import pandas as pd
from prophet import Prophet
from prophet.diagnostics import cross_validation
df = pd.read_csv('https://raw.githubusercontent.com/facebook/prophet/main/examples/example_wp_log_peyton_manning.csv')
df = df.head(1200)
m = Prophet()
m.fit(df)
# First try with initial = 730 days
df_cv1 = cross_validation(m, initial='730 days', period='180 days', horizon='90 days')
df_cv1.query("cutoff=='2010-02-15'")[['ds', 'yhat', 'cutoff']].head(3)
# ds yhat cutoff
# 2010-02-16 8.957788 2010-02-15
# 2010-02-17 8.724250 2010-02-15
# 2010-02-18 8.608022 2010-02-15
# Second try with initial = 400 days
df_cv2 = cross_validation(m, initial='400 days', period='180 days', horizon='90 days')
df_cv2.query("cutoff=='2010-02-15'")[['ds', 'yhat', 'cutoff']].head(5)
# ds yhat cutoff
# 2010-02-16 8.957788 2010-02-15
# 2010-02-17 8.724250 2010-02-15
# 2010-02-18 8.608022 2010-02-15
# Same results, even though we expected the training periods to be different
```
I've also checked the related source code, `initial` is not taken into account when defining training data :
https://github.com/facebook/prophet/blob/e665430adcd7690a1ea7565803f34043596045fe/python/prophet/diagnostics.py#L230-L239
Is this behaviour intended ?
Also wouldn't it make more sense that the folds in cross validation have a fixed size instead (rolling window of x days prior to the cutoff) ?
Contributor guide
Assessment
This issue has not been assessed yet.