cmu-delphi / cmu-delphi/forecast-eval

Consider tracking inclusion on a per-forecast basis

Open
#74 0 comments 0 reactions 0 assignees View on GitHub
data-pipeline
Dominant language
R
Stars
6
Forks
3
PR merge metrics
No merged PRs in 30d

Description

While looking into memory and runtime of the predictions cards script, I noticed that we pull a lot more data incrementally than I would've expected:

```
[1] "Out of 2026 previous forecasts we will pull 664 new forecasts\n"
1/78:Auquan-SEIR with 12 forecasts ...
2/78:CovidAnalytics-DELPHI with 11 forecasts ...
3/78:UMass-MechBayes with 2 forecasts ...
4/78:CMU-TimeSeries with 0 forecasts ...
5/78:Columbia_UNC-SurvCon with 1 forecasts ...
6/78:Geneva-DetGrowth with 19 forecasts ...
7/78:MIT_CritData-GBCF with 2 forecasts ...
8/78:UA-EpiCovDA with 5 forecasts ...
9/78:UCSB-ACTS with 0 forecasts ...
10/78:GT_CHHS-COVID19 with 7 forecasts ...
11/78:Imperial-ensemble2 with 0 forecasts ...
12/78:JHUAPL-Bucky with 0 forecasts ...
13/78:UCSD_NEU-DeepGLEAM with 0 forecasts ...
14/78:Caltech-CS156 with 0 forecasts ...
15/78:UCF-AEM with 0 forecasts ...
16/78:FAIR-NRAR with 82 forecasts ...
```

I suspect that these are all forecasts that are filtered out due to our inclusion criteria after everything has been pulled down. We should consider tracking forecasts separately on if they were included. Not only would we be able to surface more detail on specific forecasters on the front-end, but we wouldn't have to pull down previously filtered forecasts every time the prediction cards are updated.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by locating the predictions cards script and trace how forecasts are pulled and filtered by the inclusion criteria. Determine where per-forecast inclusion could be recorded, then verify that updating prediction cards avoids refetching previously filtered forecasts and can expose the additional forecaster detail described in the issue.

Written by the indexing model from the issue text.

Assessment

Tech stack
r
Domain
data
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.