Scale SLURM memory requests to network size; measure them with a benchmark sweep

Problem

SLURM memory requests are not commensurate with the size of the network being built or solved. mem_mb is either a flat constant that ignores every wildcard, or a heuristic keyed off input file bytes with a coefficient nobody has re-fit since v0. Under-requesting is a hard OOM kill; over-requesting costs queue priority and allocation. Right now we do both, in different rules, for no principled reason.

Companion to #808, which covers stale/over-provisioned walltime and the missing benchmark: directives. This one is specifically about memory vs. network size.

Current state on origin/develop

Of ~59 rules:

• 37 request a flat constant independent of interconnect, clusters, simpl, snapshot resolution, planning horizons, or sector coupling. A usa build at 500 nodes with 8760 hourly snapshots and a texas build at 10 nodes with 24 snapshots ask sbatch for exactly the same memory.
• 17 scale as input.size // N * attempt * K — the byte size of the input files. For compressed netCDF this is a weak proxy for peak RSS, and the divisors/multipliers (//70000 * 2, //150000 * 1.5, //100000 * 150) are unexplained magic numbers.

Illustrative inconsistencies:

Two structural gaps:

1. No config layer for memory. walltime: lives in config.slurm.yaml and is per-rule overridable (config_provider("walltime", "<rule>")). mem_mb has no equivalent — tuning it for a site means editing workflow/rules/*.smk.
2. memory(w) in workflow/rules/common.smk:93 is dead code. It is the PyPSA-Eur size-aware estimator (15000 + 195 * clusters, scaled by the Nh/Nseg opts and the number of planning horizons) and it is never called from anywhere in the workflow. Either wire up something like it or delete it.

Proposal

1. Measure first — run a sizing sweep. Nothing here should be guessed. Build and solve a matrix that spans the dimensions we believe drive memory, with benchmark: on every rule (#808 item 1) so each job emits max_rss / max_vms / s:

A fractional design (not the full cross product) is enough: hold everything else fixed while sweeping one axis at a time, plus a couple of corner cases (usa × 500 × 1h × E-G) to check the fit doesn't blow up at the top end.

2. Fit per-rule memory to a structural regressor, not to file bytes. Candidate regressors:

• build/cluster rules: n_buses, n_generators, number of bus regions, cutout cell count
• solve_network: LP size ≈ (n_gen + n_lines + n_storage) × n_snapshots × n_horizons; better still, log the actual variable/constraint/nonzero counts the solver reports and fit against those
• sector rules: the above × number of sector carriers

Target form mem_mb = a + b * regressor, fit from the sweep, with asymmetric headroom — memory overrun kills the job, so ~1.5× on fitted peak RSS, and keep the existing attempt multiplier as retry escalation.

3. Add a mem: block to config.slurm.yaml mirroring walltime:, so rules read config_provider("mem", "<rule>", default=<fitted expression>) and sites can override per rule without touching workflow code.

4. Ship the aggregation script proposed in #808 item 3 so it reports recommended mem_mb alongside walltime from accumulated benchmarks/ TSVs, and re-fitting is a one-command job as the model grows.

Open questions

• Do we want one fitted expression per rule, or a small number of rule classes (retrieve / build / cluster / solve) with shared coefficients? The latter is far less to maintain and probably accurate enough for everything except solve_network.
• Sweep cost: a usa × 500 × 1h solve is expensive to run purely for measurement. Do we fit the solve rule from runs we're doing anyway (opportunistic benchmark collection) rather than a dedicated sweep?

Read workflow/rules/common.smk:93 and config.slurm.yaml first, then trace the current mem_mb expressions in workflow/rules/*.smk. Use the proposed benchmark sweep and accumulated benchmarks/ TSVs to compare max_rss, max_vms, and runtime against network-size regressors; done means fitted sizing, configurable mem overrides, and aggregation output.