eduaguilera / eduaguilera/whep
[SOC #123] Turnkey runs dominated by full-range build_primary_production + global gridding/livestock, not the SOC engine
- Dominant language
- R
- Stars
- 1
- Forks
- 5
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 186
Description
Profiled from the first turnkey version in #123, and confirmed by a completed
real-data run.
## Finding
For a turnkey `build_carbon_balance()`, the cost is almost entirely in
**assembling the carbon inputs**, not in the SOC engine:
- A real Iberia-scoped end-to-end run: input read **768 s**, SOC engine
(equilibrium + march + finalise) only **166 s**.
- The read is dominated by `get_primary_production()` → `build_primary_production()`
which **always rebuilds the full 1850-2023 global series** regardless of the
requested `years` (the year filter is applied *after*). Scoping it to
2000-2003 alone builds in ~105 s vs. the full-range slog.
- The remaining read cost is the **global** manure `build_livestock_nutrient_flows()`
+ the national→grid gridding in `.sci_to_grid()`, which do not shrink with the
year window (spatial, not temporal).
This **supersedes the carbon-path assumption in #353**: that issue targeted the
SWC/water read path, but that path is already fast (SWC read ~6 s via the
existing `ncvar_get(start=, count=)` slice), and the `.cb_march` O(cells^2) item
from #353 is already fixed in this PR.
## Directions (in payoff order)
1. **Year-aware `get_primary_production()`**: thread the requested year range into
`build_primary_production(start_year=, end_year=)` and put the range in the
cache key. Guard the historical back-cast (truncating `start_year` must not
silently change extension results). Cross-cutting: every caller shares the
`primary_prod` cache, so this needs care.
2. **Disk-persist `.build_cache`** (primary_prod / cbs_built) across sessions, so
the expensive build happens once rather than per session. Needs a
staleness/invalidation design.
3. **Spatial windowing** for `build_soil_carbon_inputs` / `.sci_to_grid` so a
regional request does not pay global gridding + livestock-flow cost.
## Notes
- Not SOC-specific: `build_primary_production` / `get_wide_cbs` are shared package
infrastructure (footprint + every extension). Best done in its own PR with a
before/after timing + identical-output check, not as a rider on #123.
- Engine correctness is unaffected and already verified end-to-end on real data.
Contributor guide
Assessment
This issue has not been assessed yet.