tidyverts / tidyverts/fable

fable vs. forecast ETS model estimation performance

Open
#366 3 comments 0 reactions 1 assignee View on GitHub

@mitchelloharawild is already working on this.

Since Apr 13, 2022.

improvement
Dominant language
R
Stars
588
Forks
71
Avg merge
1h 58m
Merged PRs (30d)
1

Description

I'm seeing a large difference in the performance of the forecast ets function and the fable ETS function when estimating multiple models, surprisingly in favor of forecast. Here's a simple example using the "tourism" data set from fpp3. The inelegant loop in the forecast section is intentional. Apologies for the rest of the code in advance:

# libraries ---------------------------------------------------------------

library(fpp3)
library(forecast)

# fable functions ---------------------------------------------------------

start_time <- Sys.time()

fable_fit <- tourism %>%
  model(
    ets = ETS(Trips)
  )

fable_out <- fable_fit %>%
  forecast(h=4) %>%
  hilo(c(95,80)) %>%
  unpack_hilo(c('95%','80%')) %>%
  as_tibble() %>%
  unite(Key_FC,Region, State, Purpose, Quarter, sep = '@', remove = FALSE)

end_time <- Sys.time()

fable_time <- end_time - start_time
cat("Time for `fable` estimation & forecast:", round(as.numeric(fable_time,units = "secs"),2), "sec\n")


# forecast functions ------------------------------------------------------
tourism_aug <- tourism %>%
    unite(Key,Region, State, Purpose, sep = '@', remove = TRUE)

keys <- unique(tourism_aug$Key)

forecast_out <- tibble()

start_time <- Sys.time()
for(i in 1:length(keys)) {
  ts_single <- tourism_aug %>% 
                filter(Key == keys[i]) %>% 
                select(-Key) %>%
                ts(frequency=4, start=c(1998,1))
  
  forecast_new <- forecast(ets(ts_single[,2]),h=4) %>%
    as_tibble(rownames="index") %>%
    mutate(Key = keys[i])
  
  forecast_out <- forecast_out %>% bind_rows(forecast_new)
}
end_time <- Sys.time()

forecast_time <- end_time - start_time
cat("Time for `forecast` estimation & forecast:", round(as.numeric(forecast_time,units = "secs"),2), "sec\n")


# Comparison --------------------------------------------------------------

comparison_data <- forecast_out %>% 
  unite(Key_FC, Key, index, sep="@", remove = FALSE) %>%
    left_join(fable_out,
            by = c("Key_FC" = "Key_FC"),
            suffix = c(".forecast",".fable")) %>%
  mutate(mean_2 = (`Point Forecast` - .mean)^2) %>%
  mutate(`80%_lower_2` = (`80%_lower`-`Lo 80`)^2) %>%
  mutate(`80%_upper_2` = (`80%_upper`-`Hi 80`)^2) %>%
  mutate(`95%_lower_2` = (`95%_lower`-`Lo 95`)^2) %>%
  mutate(`95%_upper_2` = (`95%_upper`-`Hi 95`)^2) %>%
  arrange(desc(mean_2))

comparison_summary <- comparison_data %>%
  select(Key, Quarter,
                mean_2, `80%_lower_2`, `80%_upper_2`,
                `95%_lower_2`,`95%_upper_2`) %>%
  group_by(Key) %>%
  summarise(MSE_mean = mean(mean_2),
            `MSE_80%_lower` = mean(`80%_lower_2`),
            `MSE_80%_upper` = mean(`80%_upper_2`),
            `MSE_95%_lower` = mean(`95%_lower_2`),
            `MSE_95%_upper` = mean(`95%_upper_2`),
            max_err = max(MSE_mean,
                          `MSE_80%_lower`,
                          `MSE_80%_upper`,
                          `MSE_95%_lower`,
                          `MSE_95%_upper`
            )
  ) %>%
  arrange(desc(max_err))

On my machine, the output is

Time for `fable` estimation & forecast: 106.93 sec
Time for `forecast` estimation & forecast: 37.03 sec

The comparison_summary table seems to indicate that the models are giving the same forecasts, only fable is taking almost 3 times as long as using forecast in a simple loop.

Am I using either of the functions incorrectly? My first thought was that fable ETS is searching a much larger set of models, but on default settings the search space for both algorithms should be the same.

I've cross-posted this to StackOverflow in case this is just a simple error or misunderstanding on my part. If I get an answer there I will close the issue.

Thanks for the great library!

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.