tidyverts / tidyverts/fable

Feature request: consistent validation of test set

Open
#288 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
R
Stars
588
Forks
71
Avg merge
1h 58m
Merged PRs (30d)
1

Description

Dear colleagues,

I have trained a forecasting model for a grouped dataframe as follows:

model <- testing_data  %>% filter(a==70, b!=3) %>% filter(year(snsr_dt)<2019) %>% 
    model(prophet3 = fable.prophet::prophet(snsr_val_clean ~ season("month", 3, type = "multiplicative")),
          prophet4 = fable.prophet::prophet(snsr_val_clean ~ growth("linear") + season("week", 2, type = "multiplicative") + season("month", 2, type = "multiplicative")),
          prophet5 = fable.prophet::prophet(snsr_val_clean ~ growth("linear") + season("week", 2, type = "multiplicative") + season("year", 2, type = "multiplicative")),
          prophet6 = fable.prophet::prophet(snsr_val_clean ~ growth("linear") + season("week", 2, type = "multiplicative") + season("month", 2, type = "multiplicative") + season("year", 2, type = "multiplicative")))
fc <- model %>% forecast(h=52, testing_data  %>% filter(a==70, b!=3) %>% filter(year(snsr_dt)>2019)) 

And I asses the accuracy as follows:

test <- accuracy(fc, testing_data  %>% 
filter(a==70,b!=3) %>% 
filter(year(snsr_dt)>2019))

And I am getting the following warning (nice that this is so transparent btw):

Warning message:
The future dataset is incomplete, incomplete out-of-sample data will be treated as missing. 
104 observations are missing between 2019-01-01 and 2019-12-23 

However this presents a problem in the sense that most of the time series they do not have equal length. For processing my data as was using:

ts_tibble <- as_tsibble(df, 
                            key=c(a,b,c), index = snsr_dt)
    print("Filling gaps for not breaking groups")
    ts_tibble <- ts_tibble %>% fill_gaps()

Which definitely helped for the training of multiple models but should I understand that with the current API I need to align my timeseries to the same end_date to be able to assess the accuracy? Is there any way of making this more consistent?

BR
/Edgar

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The issue names the R entry points as as_tsibble(), fill_gaps(), forecast(), and accuracy(), but no repository files or tests. Start by reproducing the grouped-data example and tracing how accuracy() handles incomplete future data; done should be a documented or implemented consistent validation behavior for unequal series lengths.

Written by the indexing model from the issue text.

Assessment

Tech stack
r
Domain
data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.