eduaguilera / eduaguilera/whep

[SOC #123] Turnkey runs dominated by full-range build_primary_production + global gridding/livestock, not the SOC engine

Open
#367 3 comments 0 reactions 1 assignee Claimed by @eduaguilera View on GitHub
area:data-io area:production area:soc enhancement mechanical priority:medium
Dominant language
R
Stars
1
Forks
5
Avg merge
1d 11h
Merged PRs (30d)
186

Description

Profiled from the first turnkey version in #123, and confirmed by a completed
real-data run.

## Finding
For a turnkey `build_carbon_balance()`, the cost is almost entirely in
**assembling the carbon inputs**, not in the SOC engine:

- A real Iberia-scoped end-to-end run: input read **768 s**, SOC engine
(equilibrium + march + finalise) only **166 s**.
- The read is dominated by `get_primary_production()` → `build_primary_production()`
which **always rebuilds the full 1850-2023 global series** regardless of the
requested `years` (the year filter is applied *after*). Scoping it to
2000-2003 alone builds in ~105 s vs. the full-range slog.
- The remaining read cost is the **global** manure `build_livestock_nutrient_flows()`
+ the national→grid gridding in `.sci_to_grid()`, which do not shrink with the
year window (spatial, not temporal).

This **supersedes the carbon-path assumption in #353**: that issue targeted the
SWC/water read path, but that path is already fast (SWC read ~6 s via the
existing `ncvar_get(start=, count=)` slice), and the `.cb_march` O(cells^2) item
from #353 is already fixed in this PR.

## Directions (in payoff order)
1. **Year-aware `get_primary_production()`**: thread the requested year range into
`build_primary_production(start_year=, end_year=)` and put the range in the
cache key. Guard the historical back-cast (truncating `start_year` must not
silently change extension results). Cross-cutting: every caller shares the
`primary_prod` cache, so this needs care.
2. **Disk-persist `.build_cache`** (primary_prod / cbs_built) across sessions, so
the expensive build happens once rather than per session. Needs a
staleness/invalidation design.
3. **Spatial windowing** for `build_soil_carbon_inputs` / `.sci_to_grid` so a
regional request does not pay global gridding + livestock-flow cost.

## Notes
- Not SOC-specific: `build_primary_production` / `get_wide_cbs` are shared package
infrastructure (footprint + every extension). Best done in its own PR with a
before/after timing + identical-output check, not as a rider on #123.
- Engine correctness is unaffected and already verified end-to-end on real data.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.