cmu-delphi / cmu-delphi/forecast-eval
Consider tracking inclusion on a per-forecast basis
- Dominant language
- R
- Stars
- 6
- Forks
- 3
- PR merge metrics
- No merged PRs in 30d
Description
While looking into memory and runtime of the predictions cards script, I noticed that we pull a lot more data incrementally than I would've expected:
```
[1] "Out of 2026 previous forecasts we will pull 664 new forecasts\n"
1/78:Auquan-SEIR with 12 forecasts ...
2/78:CovidAnalytics-DELPHI with 11 forecasts ...
3/78:UMass-MechBayes with 2 forecasts ...
4/78:CMU-TimeSeries with 0 forecasts ...
5/78:Columbia_UNC-SurvCon with 1 forecasts ...
6/78:Geneva-DetGrowth with 19 forecasts ...
7/78:MIT_CritData-GBCF with 2 forecasts ...
8/78:UA-EpiCovDA with 5 forecasts ...
9/78:UCSB-ACTS with 0 forecasts ...
10/78:GT_CHHS-COVID19 with 7 forecasts ...
11/78:Imperial-ensemble2 with 0 forecasts ...
12/78:JHUAPL-Bucky with 0 forecasts ...
13/78:UCSD_NEU-DeepGLEAM with 0 forecasts ...
14/78:Caltech-CS156 with 0 forecasts ...
15/78:UCF-AEM with 0 forecasts ...
16/78:FAIR-NRAR with 82 forecasts ...
```
I suspect that these are all forecasts that are filtered out due to our inclusion criteria after everything has been pulled down. We should consider tracking forecasts separately on if they were included. Not only would we be able to surface more detail on specific forecasters on the front-end, but we wouldn't have to pull down previously filtered forecasts every time the prediction cards are updated.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by locating the predictions cards script and trace how forecasts are pulled and filtered by the inclusion criteria. Determine where per-forecast inclusion could be recorded, then verify that updating prediction cards avoids refetching previously filtered forecasts and can expose the additional forecaster detail described in the issue.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- r
- Domain
- data
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100