Make rules/functions modular and reusable
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 615
- Forks
- 459
- Avg merge
- 4d 14h
- Merged PRs (30d)
- 6
Description
When planning on reusing of parts of pypsa-eur in a modular fashion, user quickly face limitations. The following proposal serve as a starting point for a discussion how to make the workflow more "distributable" allowing to re-use rules and functions inside and outside of forks.
CORE IDEA: For each script create one high-level function which is runs independent of the snakemake object, has a clear signature and is importable.
Take build_renewable_profiles as an example. This could be easily restructured as:
# build_renewable_profiles.py (same file)
def build_renewable_profiles(cutout, availability_matrix, params, client=None) -> xr.Dataset:
... # hoisted out of the __main__ block; returns the Dataset
return ds
if __name__ == "__main__":
if "snakemake" not in globals():
from scripts._helpers import mock_snakemake
snakemake = mock_snakemake("build_renewable_profiles", ...)
ds = build_renewable_profiles(cutout, avail, dict(snakemake.params), client=...)
ds.to_netcdf(snakemake.output.profile)
Downstream then does from scripts.build_renewable_profiles import build_renewable_profiles.
In order to make this consistent we need to
- not leak the
snakemakeobject into the function. The function must take real arguments (cutout, matrix, a params dict/dataclass, plain paths) so it is callable without a internal dependency on the snakemake object. - define proper return values and not preempt the writing of the output.**. Keep dependency on relative paths low.
Another benefit is that pure functions are unit-testable without running Snakemake.
I iterated with Claude on this on details for time series, see below
Scan signals (counts of matching lines per script):
| Script | atlite | dask | ext. download | xarray | Tier |
|---|---|---|---|---|---|
| build_renewable_profiles | 13 | 7 | – | 8 | A |
| build_hydro_profile | 9 | – | – | 1 | A |
| build_temperature_profiles | 8 | 4 | – | 4 | A |
| build_solar_thermal_profiles | 7 | 3 | – | 3 | A |
| build_daily_heat_demand | 7 | 3 | – | 3 | A |
| build_line_rating | 7 | 9 | – | 5 | A |
| build_country_runoff | 4 | – | – | – | A |
| build_cop_profiles (pkg) | ✓ | – | – | ✓ | A (done) |
| build_hourly_heat_demand | – | – | – | 4 | B |
| build_transport_demand | – | – | – | 2 | B |
| build_electricity_demand_base | – | – | – | 5 | B |
| build_electricity_demand | – | – | ENTSO-E | – | C |
| build_mobility_profiles | – | – | ✓ | – | C |
| build_monthly_prices | – | – | ✓ | – | C |
| build_co2_prices | – | – | ✓ | – | C |
| build_shipping_demand | – | – | ✓ | – | C |
Tier A — cutout/atlite transforms → best fit
All share one signature: (cutout, clustered_regions/layout, params) → xarray.Dataset.
Deterministic functions of the cutout, no runtime network calls.
One design decision: the dask client. build_line_rating and
build_renewable_profiles construct and shutdown() the client inside the script.
Lifting as-is would couple every importer to a dask-cluster lifecycle. Make it an
injected, optional argument (client: Client | None = None) so the caller owns
parallelism.
Tier B — pure reshapers → trivial fit
No atlite, no download — dataframe/xarray reshaping only (daily→hourly,
profiles→demand, spatial distribution). Nearly pure already; only the
snakemake.input/output lines need peeling off into the shell.
Tier C — retrieval-coupled → fit only after splitting retrieve from transform
The download is half the script and is the part that is non-deterministic, needs API
keys, and must not live in an importable function. Move the download behind a thin
retrieve_* adapter; the parse→tidy→reindex-to-snapshots half becomes the importable
function. Same retrieve/transform discipline build_cop_profiles already demonstrates.
Proposed next steps (smallest valuable first)
- Add a high-level function to
build_renewable_profilesandbuild_line_rating
(in-script), injecting the dask client. Highest value, and they exercise the
dask-injection decision. - Same for Tier B scripts — mechanical, low-risk.
- For Tier C, split retrieve vs. transform; expose the transform as an in-script
function, leave retrieval behindretrieve_*.
Optional later consolidation (non-breaking)
Once the in-script functions exist, a curated pypsa_eur/ package can re-export
them for a stable public API without touching call sites:
# pypsa_eur/weather_profiles.py
from scripts.build_renewable_profiles import build_renewable_profiles
from scripts.build_line_rating import build_line_rating
...
This is the half-step the in-script approach leaves open: it fully serves the
soft-fork / branch-off case today (we already vendor the repo and do
from scripts.gb_model._helpers import ...), and the pip-installable-library case
becomes a later re-export layer rather than a prerequisite refactor.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading scripts/build_renewable_profiles.py and the build_line_rating script, then compare their Snakemake entry points with the existing build_cop_profiles pattern. The first milestone is importable high-level functions with explicit arguments, return values, and optional client injection; completion should cover the proposed Tier A and later Tier B/C workflow without leaking Snakemake into the functions.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- build-system, tooling
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100