Confusion about test accuracy
Nobody has claimed this yet.
- Dominant language
- R
- Stars
- 588
- Forks
- 71
- Avg merge
- 1h 58m
- Merged PRs (30d)
- 1
Description
On https://www.mitchelloharawild.com/blog/fable/
Mr. Ohara-Wild writes in his blog: '[...] The in-sample accuracy suggests that the ETS model performs best. This is because it has the lowest values for all accuracy measures (lower values indicate less errors). [...]'
If I compute row means from the resulting object of accuracy() and arrange it descending, I get an opposite ranking. Models manually top ranked- are then poor for fable (in- and out of sample). What is the reason for that?
Are there any additional, internal criteria for the ranking?
bestModel<-fabletools::accuracy(myModels)
msg<-paste('Best Model Fable:')
print(msg)
info<-bestModel %>%
dplyr::rowwise() %>% mutate(myMean = mean(c(ME,RMSE,MAE,MPE,MAPE,MASE,RMSSE,ACF1),na.rm=T)) %>%
ungroup()
print(info)
bestModel<-bestModel %>% dplyr::arrange(myMean)
msg<-paste('Best Model Ich:')
print(msg)
print(bestModel)
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the accuracy() output and the fabletools model-selection behavior described in the issue, then compare both rankings against the linked blog explanation. Done means identifying whether the discrepancy comes from the ranking criteria or the example and documenting the reason clearly.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- r
- Domain
- data
- Issue type
- Documentation
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100