Feature request: consistent validation of test set
Nobody has claimed this yet.
- Dominant language
- R
- Stars
- 588
- Forks
- 71
- Avg merge
- 1h 58m
- Merged PRs (30d)
- 1
Description
Dear colleagues,
I have trained a forecasting model for a grouped dataframe as follows:
model <- testing_data %>% filter(a==70, b!=3) %>% filter(year(snsr_dt)<2019) %>%
model(prophet3 = fable.prophet::prophet(snsr_val_clean ~ season("month", 3, type = "multiplicative")),
prophet4 = fable.prophet::prophet(snsr_val_clean ~ growth("linear") + season("week", 2, type = "multiplicative") + season("month", 2, type = "multiplicative")),
prophet5 = fable.prophet::prophet(snsr_val_clean ~ growth("linear") + season("week", 2, type = "multiplicative") + season("year", 2, type = "multiplicative")),
prophet6 = fable.prophet::prophet(snsr_val_clean ~ growth("linear") + season("week", 2, type = "multiplicative") + season("month", 2, type = "multiplicative") + season("year", 2, type = "multiplicative")))
fc <- model %>% forecast(h=52, testing_data %>% filter(a==70, b!=3) %>% filter(year(snsr_dt)>2019))
And I asses the accuracy as follows:
test <- accuracy(fc, testing_data %>%
filter(a==70,b!=3) %>%
filter(year(snsr_dt)>2019))
And I am getting the following warning (nice that this is so transparent btw):
Warning message:
The future dataset is incomplete, incomplete out-of-sample data will be treated as missing.
104 observations are missing between 2019-01-01 and 2019-12-23
However this presents a problem in the sense that most of the time series they do not have equal length. For processing my data as was using:
ts_tibble <- as_tsibble(df,
key=c(a,b,c), index = snsr_dt)
print("Filling gaps for not breaking groups")
ts_tibble <- ts_tibble %>% fill_gaps()
Which definitely helped for the training of multiple models but should I understand that with the current API I need to align my timeseries to the same end_date to be able to assess the accuracy? Is there any way of making this more consistent?
BR
/Edgar
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The issue names the R entry points as as_tsibble(), fill_gaps(), forecast(), and accuracy(), but no repository files or tests. Start by reproducing the grouped-data example and tracing how accuracy() handles incomplete future data; done should be a documented or implemented consistent validation behavior for unequal series lengths.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- r
- Domain
- data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100