(loo_)(s)crps could ask for only one argument with predictions
Nobody has claimed this yet.
- Dominant language
- R
- Stars
- 157
- Forks
- 38
- Avg merge
- 4d 16h
- Merged PRs (30d)
- 2
Description
I think it would make sense to ask the user for only one argument with predictions with at least two columns and split the predictions array. (Also in #221).
In my tests, if you just divide the predictions in two, the estimates are virtually the same:
(ps2 <- crps(ypred1[1:2000,], ypred1[2001:4000,], y = fit$y))
@avehtari said:
I think it would be clearer to have this discussed in a separate issue (and tag @LeeviLindgren). I don't expect there is much difference in the easy case, but if the variability of the importance weights is big then a) the importance weights are different for the two halves (while in the current implementation they are the same), b) the number of draws for the importance weighting is halved, which increases Monte Carlo error. It is possible that in practice a + b are not a problem, but this should be investigated more thoroughly.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at the crps entry point and review the issue #221 discussion alongside the current behavior for separate prediction arguments. Compare split and combined predictions, focusing on how importance weights and the number of draws affect Monte Carlo error. Done means the proposed one-argument interface has been investigated and its behavior is supported by tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- r
- Domain
- data
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100