tidyverts / tidyverts/fabletools
Different results with and without library(future)
Nobody has claimed this yet.
- Dominant language
- R
- Stars
- 98
- Forks
- 37
- PR merge metrics
- No merged PRs in 30d
Description
When I load package future before running my modelling code, the models (and forecasts) produced with the tidy fabletools workflow differ from the models (and forecasts) produced by the underlying base functions in echos. Without future the two approaches return identical numbers.
Hint: The functions ESN() and forecast.ESN() are just wrapper for train_esn() and forecast_esn() and should produce the same results. There is randomization within the ESN model handled internally by set.seed().
Minimal reproducible example
# Case (1) WITHOUT library(future) ============================================
library(echos)
library(fabletools)
library(tidyverse)
# Case (1.a) (base function) --------------------------------------------------
xa <- as.numeric(AirPassengers)
modela <- train_esn(xa)
summary(modela)
fcsta <- forecast_esn(modela, n_ahead = 18)
fcsta <- fcsta$point
# Case (1.b) (tidy function) --------------------------------------------------
xb <- as_tsibble(AirPassengers)
modelb <- xb %>%
model("ESN" = ESN(value))
report(modelb)
fcstb <- modelb %>% forecast(h = 18)
fcstb <- fcstb$.mean
# Comparison ------------------------------------------------------------------
all.equal(fcsta, fcstb)
# Case (2) WITH library(future) ===============================================
library(echos)
library(fabletools)
library(tidyverse)
library(future)
# Case (2.a) (base function) --------------------------------------------------
xa <- as.numeric(AirPassengers)
modela <- train_esn(xa)
summary(modela)
fcsta <- forecast_esn(modela, n_ahead = 18)
fcsta <- fcsta$point
# Case (2.b) (tidy function) --------------------------------------------------
xb <- as_tsibble(AirPassengers)
modelb <- xb %>%
model("ESN" = ESN(value))
report(modelb)
fcstb <- modelb %>% forecast(h = 18)
fcstb <- fcstb$.mean
# Comparison ------------------------------------------------------------------
all.equal(fcsta, fcstb)
Question
I think the issue begins within model()... Is model() and/or forecast() (or other fabletools internals) expected to react differently when future is loaded? If not, could there be an interaction with future’s RNG handling that slips through the tidy interface but not the base one?
Any tips or recommended work-arounds would be much appreciated.
Thank you for your time and for the fantastic package!
Best regards,
Alex
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the minimal reproducible example and compare the results of train_esn()/forecast_esn() with the ESN()/forecast() workflow, both with and without library(future). Read the model() and forecast() paths mentioned in the report, then determine whether the workflows should produce identical values and document the expected result or workaround.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- r
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100