Use a stable cache for input data

Describe the feature you'd like to see

Harden input data retrieval steps by using a stable cache such as data.pypsa.org for all workflow data inputs not version-controlled via git.

Motivation

We do use large parts from the PyPSA-DE workflow in PyPSA-AT and run tests frequently on private pipelines and on public github actions. If remote files are not available from remote resources, the snakemake workflow fails early. This is the case even for data files that already exist in a local cache, because the retrieval rules check for the existence of remote resources regardless.

Another pro for using a cache is reproducibility: Input files may change e.g. the .xlsx files, and older results may turn irreproducible on input data updates.

List of resources I know of that sometimes fail

• https://geodata.ucdavis.edu/gadm/gadm4.1/json/gadm41_DEU_1.json.zip
• https://fnb-gas.de/wp-content/uploads/2024/07/2024_07_22_Anlage3_FNB_Massnahmenliste_Neubau.xlsx
• https://fnb-gas.de/wpcontent/uploads/2024/07/2024_07_22_Anlage2_Leitungsmeldungen_weiterer_potenzieller_Wasserstoffnetzbetreiber.xlsx
• https://fnb-gas.de/wp-content/uploads/2024/07/2024_07_22_Anlage4_FNB_Massnahmenliste_Umstellung.xlsx
• https://pubdb.bfe.admin.ch/de/publication/download/12361

Are there reasons for the resources not being copied to a cache? Maybe licensing restricts storing copies?

Review the Snakemake retrieval rules and the listed remote resources first, then determine which non-git inputs need stable cached copies and whether licensing permits redistribution. Done means workflow inputs remain reproducible and local cached files can be used when the original remote resources are unavailable.