Re-organizing and cleaning up this codebase

We've discussed in group meetings that this codebase needs a bit of re-organization and cleanup work. With AI assisted development, our small team might have a chance at actually making this happen.

- [ ] A major first push came here #208. We should follow-up on the comments in the review there.
- [ ] The same PR added `CLAUDE.md` and `notes/` which contain further possible follow ups

Some general thoughts for directing future work.

---

I've been thinking a bit about how to improve these types of ML codebases. Having worked on a few, I've seen the patterns that tend to come up and the ways in which code degrades. I'm going to see if I can synthesize together a high-level vision of the codebase and see if that can guide us to something more readable, maintainable, and extendable.

What I find handy is to split the codebase into layers and consider each of those in turn:

- there's the data munging layer
- there's the forecaster implementation layer
- there's the orchestration layer

The data munging layer consists of managing large data archives. In our instance, this tends to involve
- pulling the data from upstream (ideally from our own APIs, but sometimes from disparate sources, if the data is not available there),
- organizing and normalizing this data into a common schema,
- performing date-alignment (some sources are at the epiweek level or simply have different conventions for weeks, e.g. starting on Wednesdays vs Sundays)
- munging: trimming outliers, trimming low information geos, recoding non-standard named geos
- managing the archive revisions history, merging archive histories (e.g. using our own epix_merge)
- feature-creation (seasonal information, population, exogenous signals)
- units/scaling (per 100k,
- normalization/whitening (this sort of thing actually takes place in the forecaster, but including here since it's thematically relevant)
- etc.

The forecaster implementation layer involves
- defining a forecaster input expectations: epi_df, epi_archive, latest or cheating, etc.
- defining whether a forecaster is one shot at a given ahead or is iterative or it does them all at once
- performs date alignment of the time series to create lagged features
- selects exogenous features from the dataframe provided
- determines whether to geo-pool when training or not
- what scale does the forecaster expect the input to be on vs what scale does it emit
- trims the dataset to avoid nulls (or other)
- whitening/scaling (see above)
- eventually produces the data matrix for the fitting problem which is fed to the optimizer / fitting function
- postprocesses the data (e.g. undoes scaling if needed)

The orchestration layer involves
- keeping track of the DAG of tasks that need to be run to produce forecasters (roughly: get data, build archives, feed the right data to forecasters, feed the right forecasts to the right ensembles, produce audit artifacts like notebooks, and produce submission files)
- caching all the intermediate outputs of these tasks; invalidating cache appropriately (data changed or code changed)
- aids in debugging by allowing introspection of each intermediate task output
- for us, the targets framework is what handles all this

---

Challenges (working observations, still in draft):
- a major challenge for newcomers is targets' rather opaque execution model; understanding this model is necessary to find the right entry point to introduce a new forecaster or a new dataset; this can probably be improved with clear guides on how to do this (documentation) and also by sharpening up the contracts between the different layers
- speaking of contracts, this is one of the core guiding principles of #208 and we should continue with it; the idea is to place assertions in key places of the pipeline to make implicit assumptions explicit (e.g. about expected columns, about expected reference time rows, about expected data scale, etc.); there are frameworks like [Pandera](https://pandera.readthedocs.io/en/latest/index.html) that we can take inspiration from
- it already arises in my descriptions above, but data and forecaster are fundamentally linked and can't be fully separated
- but this partitioning helps push for a principle - by consolidating data processing to a single consistent place we can prevent bugs arising from inconsistent processing
- sharing code is generally good; the hard part is finding the right abstractions so we can make reusing code easy (instead of an exercise in contortion)
- sharing code in the data layer tends to be a pretty easy win - dataframe munging operations are not too hard to generalize
- sharing code in the forecasting layer has been pretty difficult - epipredict is our package that aimed at exactly this and it ran into a number of challenges; part of it was that we built on a foundation (tidymodels) that fundamentally was not equipped for panel data, so we paid for integration with the framework, but got very little out of it, as we had to implement all their steps ourselves; another major difficulty was date alignment of lags/leads in multi-variate time series models and appropriately determining the train/test split in response to the various features each forecaster included (e.g. is smoothing, pad an extra N days)
- sharing code in the orchestration layer is mixed - on the one hand, to add and test a tiny forecaster, you need to run the full behemoth exploration pipeline, on the other hand, once you figure that out, you benefit from the rest of the framework

Start by reading the review comments on #208, then inspect CLAUDE.md and the follow-up material in notes/. The issue describes possible data, forecaster, and orchestration concerns but does not identify a bounded entry point, files to change, tests to run, or a concrete definition of done; agree on a smaller scoped task before starting.

Written by the indexing model from the issue text.