fable vs. forecast ETS model estimation performance
@mitchelloharawild is already working on this.
Since Apr 13, 2022.
- Dominant language
- R
- Stars
- 588
- Forks
- 71
- Avg merge
- 1h 58m
- Merged PRs (30d)
- 1
Description
I'm seeing a large difference in the performance of the forecast ets function and the fable ETS function when estimating multiple models, surprisingly in favor of forecast. Here's a simple example using the "tourism" data set from fpp3. The inelegant loop in the forecast section is intentional. Apologies for the rest of the code in advance:
# libraries ---------------------------------------------------------------
library(fpp3)
library(forecast)
# fable functions ---------------------------------------------------------
start_time <- Sys.time()
fable_fit <- tourism %>%
model(
ets = ETS(Trips)
)
fable_out <- fable_fit %>%
forecast(h=4) %>%
hilo(c(95,80)) %>%
unpack_hilo(c('95%','80%')) %>%
as_tibble() %>%
unite(Key_FC,Region, State, Purpose, Quarter, sep = '@', remove = FALSE)
end_time <- Sys.time()
fable_time <- end_time - start_time
cat("Time for `fable` estimation & forecast:", round(as.numeric(fable_time,units = "secs"),2), "sec\n")
# forecast functions ------------------------------------------------------
tourism_aug <- tourism %>%
unite(Key,Region, State, Purpose, sep = '@', remove = TRUE)
keys <- unique(tourism_aug$Key)
forecast_out <- tibble()
start_time <- Sys.time()
for(i in 1:length(keys)) {
ts_single <- tourism_aug %>%
filter(Key == keys[i]) %>%
select(-Key) %>%
ts(frequency=4, start=c(1998,1))
forecast_new <- forecast(ets(ts_single[,2]),h=4) %>%
as_tibble(rownames="index") %>%
mutate(Key = keys[i])
forecast_out <- forecast_out %>% bind_rows(forecast_new)
}
end_time <- Sys.time()
forecast_time <- end_time - start_time
cat("Time for `forecast` estimation & forecast:", round(as.numeric(forecast_time,units = "secs"),2), "sec\n")
# Comparison --------------------------------------------------------------
comparison_data <- forecast_out %>%
unite(Key_FC, Key, index, sep="@", remove = FALSE) %>%
left_join(fable_out,
by = c("Key_FC" = "Key_FC"),
suffix = c(".forecast",".fable")) %>%
mutate(mean_2 = (`Point Forecast` - .mean)^2) %>%
mutate(`80%_lower_2` = (`80%_lower`-`Lo 80`)^2) %>%
mutate(`80%_upper_2` = (`80%_upper`-`Hi 80`)^2) %>%
mutate(`95%_lower_2` = (`95%_lower`-`Lo 95`)^2) %>%
mutate(`95%_upper_2` = (`95%_upper`-`Hi 95`)^2) %>%
arrange(desc(mean_2))
comparison_summary <- comparison_data %>%
select(Key, Quarter,
mean_2, `80%_lower_2`, `80%_upper_2`,
`95%_lower_2`,`95%_upper_2`) %>%
group_by(Key) %>%
summarise(MSE_mean = mean(mean_2),
`MSE_80%_lower` = mean(`80%_lower_2`),
`MSE_80%_upper` = mean(`80%_upper_2`),
`MSE_95%_lower` = mean(`95%_lower_2`),
`MSE_95%_upper` = mean(`95%_upper_2`),
max_err = max(MSE_mean,
`MSE_80%_lower`,
`MSE_80%_upper`,
`MSE_95%_lower`,
`MSE_95%_upper`
)
) %>%
arrange(desc(max_err))
On my machine, the output is
Time for `fable` estimation & forecast: 106.93 sec
Time for `forecast` estimation & forecast: 37.03 sec
The comparison_summary table seems to indicate that the models are giving the same forecasts, only fable is taking almost 3 times as long as using forecast in a simple loop.
Am I using either of the functions incorrectly? My first thought was that fable ETS is searching a much larger set of models, but on default settings the search space for both algorithms should be the same.
I've cross-posted this to StackOverflow in case this is just a simple error or misunderstanding on my part. If I get an answer there I will close the issue.
Thanks for the great library!
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.