tidyverts / tidyverts/fabletools

Different results with and without library(future)

Open
#420 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

parellel
Dominant language
R
Stars
98
Forks
37
PR merge metrics
No merged PRs in 30d

Description

When I load package future before running my modelling code, the models (and forecasts) produced with the tidy fabletools workflow differ from the models (and forecasts) produced by the underlying base functions in echos. Without future the two approaches return identical numbers.

Hint: The functions ESN() and forecast.ESN() are just wrapper for train_esn() and forecast_esn() and should produce the same results. There is randomization within the ESN model handled internally by set.seed().

Minimal reproducible example

# Case (1) WITHOUT library(future) ============================================

library(echos)
library(fabletools)
library(tidyverse)

# Case (1.a) (base function) --------------------------------------------------
xa <- as.numeric(AirPassengers)
modela <- train_esn(xa)
summary(modela)
fcsta <- forecast_esn(modela, n_ahead = 18)
fcsta <- fcsta$point

# Case (1.b) (tidy function) --------------------------------------------------
xb <- as_tsibble(AirPassengers)
modelb <- xb %>%
  model("ESN" = ESN(value))
report(modelb)
fcstb <- modelb %>% forecast(h = 18)
fcstb <- fcstb$.mean

# Comparison ------------------------------------------------------------------
all.equal(fcsta, fcstb)
# Case (2) WITH library(future) ===============================================

library(echos)
library(fabletools)
library(tidyverse)
library(future)

# Case (2.a) (base function) --------------------------------------------------
xa <- as.numeric(AirPassengers)
modela <- train_esn(xa)
summary(modela)
fcsta <- forecast_esn(modela, n_ahead = 18)
fcsta <- fcsta$point

# Case (2.b) (tidy function) --------------------------------------------------
xb <- as_tsibble(AirPassengers)
modelb <- xb %>%
  model("ESN" = ESN(value))
report(modelb)
fcstb <- modelb %>% forecast(h = 18)
fcstb <- fcstb$.mean

# Comparison ------------------------------------------------------------------
all.equal(fcsta, fcstb)

Question
I think the issue begins within model()... Is model() and/or forecast() (or other fabletools internals) expected to react differently when future is loaded? If not, could there be an interaction with future’s RNG handling that slips through the tidy interface but not the base one?

Any tips or recommended work-arounds would be much appreciated.

Thank you for your time and for the fantastic package!

Best regards,
Alex

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the minimal reproducible example and compare the results of train_esn()/forecast_esn() with the ESN()/forecast() workflow, both with and without library(future). Read the model() and forecast() paths mentioned in the report, then determine whether the workflows should produce identical values and document the expected result or workaround.

Written by the indexing model from the issue text.

Assessment

Tech stack
r
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.