facebook / facebook/prophet

Difference in outputs from cross_validation function between daily and weekly data

Aperta
#2,005 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub
question
Lingua principale
Python
Stelle
20.4k
Fork
4.6k
Merge medio
19h 52m
PR unite (30g)
1

Descrizione

Hi all!

I noticed that the metrics results from the _cross_validation_ and _performance_metrics_ functions were not varying depending on the combination of hyperparameters. I'm coding in pyspark and using [applyInPandas](https://spark.apache.org/docs/latest/api/python/reference/api/pyspark.sql.GroupedData.applyInPandas.html) to forecast a **weekly** time series in pyspark data frame format.

So I got an example from [medium](https://higee.medium.com/parallel-model-training-with-prophet-and-spark-5f40be750f97) with a **daily** time series and the outputs from _cross_validation_ and _performance_metrics_ functions are varying.

![image](https://user-images.githubusercontent.com/46826549/130310325-18bae0a4-9a24-4818-9647-f1dd1f35a42b.png)

For a test, I resampled the same data from tutorial to weekly timestamp, and the outputs did not vary.

![image](https://user-images.githubusercontent.com/46826549/130310334-e5e3ed12-9758-4163-90ba-39517927df8a.png)

The function that I'm using to run cross validation using prophet is:

```python
# based on: https://higee.medium.com/parallel-model-training-with-prophet-and-spark-5f40be750f97
def run_cross_validation(keys, df):

# sorting through timestamp
df = df.sort_values('ds')

# setting keys
item, seasonality_mode, changepoint_prior_scale, seasonality_prior_scale = keys

# model instance
model = Prophet(
seasonality_mode=seasonality_mode,
changepoint_prior_scale=changepoint_prior_scale,
seasonality_prior_scale=seasonality_prior_scale
)

# training model
model.fit(df)

# cross-validation for daily data
df_cv = cross_validation(
model,
horizon=pd.Timedelta('{} W'.format(8)),
parallel='processes'
)

# getting performance metrics
df_perf_metrics = performance_metrics(
df_cv,
metrics=['mse'],
rolling_window=1
)

metric = np.mean(df_perf_metrics['mse'].values[:3])

df_perf_metrics_mse = pd.DataFrame(
data=[metric],
columns=['mse']
)

mapping = {
'item': item,
'seasonality_mode': seasonality_mode,
'changepoint_prior_scale': changepoint_prior_scale,
'seasonality_prior_scale': seasonality_prior_scale
}

df_perf_metrics_mse = df_perf_metrics_mse.assign(**mapping)

cols = [
'item',
'mse',
'seasonality_mode',
'changepoint_prior_scale',
'seasonality_prior_scale'
]

return df_perf_metrics_mse[cols]

df_param = spark.createDataFrame([
(
['multiplicative', 'additive'],
[0.001, 0.005, 0.01, 0.1, 0.3, 0.5],
[0.01, 0.05, 0.1, 1.0, 5.0, 10.0],
)],
[
'seasonality_mode',
'changepoint_prior_scale',
'seasonality_prior_scale'
]
)

for col in df_param.columns:
df_param = df_param.withColumn(col, explode(col))

df_input = spark.createDataFrame(df) # pandas data frame to pyspark data frame

df_input_param = df_input.crossJoin(df_param)

cross_validation_schema = StructType([
StructField('item', StringType()),
StructField('mse', FloatType()),
StructField('seasonality_mode', StringType()),
StructField('changepoint_prior_scale', FloatType()),
StructField('seasonality_prior_scale', FloatType())
])

group_by_col = [
'item',
'seasonality_mode',
'changepoint_prior_scale',
'seasonality_prior_scale'
]

df_cross_validation = df_input_param.groupBy(group_by_col).applyInPandas(func=run_cross_validation, schema=cross_validation_schema)
```

Does anyone know what I'm doing wrong?

Guida per i contributori

Apri la guida per i contributori

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.