PyPSA / PyPSA/pypsa-eur

`build_renewable_profile_offwind-dc` broken for manually built `europe-2013-era5` cutout

Open
#434 18 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
615
Forks
459
Avg merge
4d 14h
Merged PRs (30d)
6

Description

Checklist

  • I am using the current master branch or the latest release. Please indicate.
  • I am running on an up-to-date pypsa-eur environment. Update via conda env update -f envs/environment.yaml.

Describe the Bug

Attempting to build renewable profile for off-shore wind DC with build_cutout: true in a freshly cloned pypsa-eur repo fails with ValueError: operands could not be broadcast together with shapes (247,221) (57,194)

snakemake -j 8 resources/profile_offwind-dc.nc

My configuration changes are shown below - I removed the sarah cutout, replaced its usages with the era5 cutout, and set build_cutout: true.

Configuration Changes 

diff --git a/config.default.yaml b/config.default.yaml
index 0ec5e9f..2ac3bc0 100755
--- a/config.default.yaml
+++ b/config.default.yaml
@@ -31,8 +31,8 @@ enable:
   prepare_links_p_nom: false
   retrieve_databundle: true
   retrieve_cost_data: true
-  build_cutout: false
-  retrieve_cutout: true
+  build_cutout: true
+  retrieve_cutout: false
   build_natura_raster: false
   retrieve_natura_raster: true
   custom_busmap: false
@@ -97,16 +97,6 @@ atlite:
       dx: 0.3
       dy: 0.3
       time: ['2013', '2013']
-    europe-2013-sarah:
-      module: [sarah, era5] # in priority order
-      x: [-12., 45.]
-      y: [33., 65]
-      dx: 0.2
-      dy: 0.2
-      time: ['2013', '2013']
-      sarah_interpolate: false
-      sarah_dir:
-      features: [influx, temperature]
 
 renewable:
@@ -170,7 +160,7 @@ renewable:
     potential: simple # or conservative
     clip_p_max_pu: 1.e-2
   solar:
-    cutout: europe-2013-sarah
+    cutout: europe-2013-era5
     resource:
       method: pv
       panel: CSi

Error Message

The error occurs on the line exclusions = exclusions | masked_ in gis.py in atlite.

As far as I understand, the error message indicates that I'm attempting a matrix operation on matrices with incompatible sizes (presumably exclusions and masked have different sizes), but I don't understand why that happens - and why it works with a retrieved europe-2013-era5 cutout.

Full Error Message 

[Mon Oct 24 13:31:21 2022]
rule build_renewable_profiles:
    input: networks/base.nc, data/bundle/corine/g250_clc06_V18_5.tif, resources/natura.tiff, data/bundle/GEBCO_2014_2D.nc, resources/shipdensity_raster.nc, resources/country_shapes.geojson, resources/offshore_shapes.geojson, resources/regions_offshore.geojson, cutouts/europe-2013-era5.nc
    output: resources/profile_offwind-dc.nc
    log: logs/build_renewable_profile_offwind-dc.log
    jobid: 0
    benchmark: benchmarks/build_renewable_profiles_offwind-dc
    reason: Missing output files: resources/profile_offwind-dc.nc; Input files updated by another job: data/bundle/corine/g250_clc06_V18_5.tif, cutouts/europe-2013-era5.nc, resources/natura.tiff, resources/regions_offshore.geojson, data/bundle/GEBCO_2014_2D.nc, resources/shipdensity_raster.nc, resources/country_shapes.geojson, resources/offshore_shapes.geojson, networks/base.nc
    wildcards: technology=offwind-dc
    threads: 4
    resources: tmpdir=/var/folders/d2/_d71yg_j05gc_0p6ldvyf9580000gn/T, mem_mb=20000

INFO:__main__:correction_factor is set as 0.8855
INFO:__main__:Calculate landuse availabilities...
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
  File "/Users/zoltan/mambaforge/envs/pypsa-eur/lib/python3.9/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/Users/zoltan/mambaforge/envs/pypsa-eur/lib/python3.9/multiprocessing/pool.py", line 48, in mapstar
    return list(map(*args))
  File "/Users/zoltan/mambaforge/envs/pypsa-eur/lib/python3.9/site-packages/atlite/gis.py", line 534, in _process_func
    return shape_availability_reprojected(shapes.loc[[i]], *args)[0]
  File "/Users/zoltan/mambaforge/envs/pypsa-eur/lib/python3.9/site-packages/atlite/gis.py", line 509, in shape_availability_reprojected
    masked, transform = shape_availability(geometry, excluder)
  File "/Users/zoltan/mambaforge/envs/pypsa-eur/lib/python3.9/site-packages/atlite/gis.py", line 463, in shape_availability
    exclusions = exclusions | masked_
ValueError: operands could not be broadcast together with shapes (247,221) (57,194)
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/zoltan/github/pypsa-bro/.snakemake/scripts/tmpyoo6zp0l.build_renewable_profiles.py", line 286, in <module>
    availability = cutout.availabilitymatrix(regions, excluder, **kwargs)
  File "/Users/zoltan/mambaforge/envs/pypsa-eur/lib/python3.9/site-packages/atlite/gis.py", line 610, in compute_availabilitymatrix
    availability = list(pool.map(_process_func, shapes.index))
  File "/Users/zoltan/mambaforge/envs/pypsa-eur/lib/python3.9/multiprocessing/pool.py", line 364, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/Users/zoltan/mambaforge/envs/pypsa-eur/lib/python3.9/multiprocessing/pool.py", line 771, in get
    raise self._value
  File "/Users/zoltan/mambaforge/envs/pypsa-eur/lib/python3.9/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/Users/zoltan/mambaforge/envs/pypsa-eur/lib/python3.9/multiprocessing/pool.py", line 48, in mapstar
    return list(map(*args))
  File "/Users/zoltan/mambaforge/envs/pypsa-eur/lib/python3.9/site-packages/atlite/gis.py", line 534, in _process_func
    return shape_availability_reprojected(shapes.loc[[i]], *args)[0]
  File "/Users/zoltan/mambaforge/envs/pypsa-eur/lib/python3.9/site-packages/atlite/gis.py", line 509, in shape_availability_reprojected
    masked, transform = shape_availability(geometry, excluder)
  File "/Users/zoltan/mambaforge/envs/pypsa-eur/lib/python3.9/site-packages/atlite/gis.py", line 463, in shape_availability
    exclusions = exclusions | masked_
ValueError: operands could not be broadcast together with shapes (247,221) (57,194)
[Mon Oct 24 13:31:49 2022]
Error in rule build_renewable_profiles:
    jobid: 0
    output: resources/profile_offwind-dc.nc
    log: logs/build_renewable_profile_offwind-dc.log (check log file(s) for error message)

RuleException:
CalledProcessErrorin line 377 of /Users/zoltan/github/pypsa-bro/Snakefile:
Command 'set -euo pipefail;  /Users/zoltan/mambaforge/envs/pypsa-eur/bin/python3.9 /Users/zoltan/github/pypsa-bro/.snakemake/scripts/tmpyoo6zp0l.build_renewable_profiles.py' returned non-zero exit status 1.
  File "/Users/zoltan/github/pypsa-bro/Snakefile", line 377, in __rule_build_renewable_profiles
  File "/Users/zoltan/mambaforge/envs/pypsa-eur/lib/python3.9/concurrent/futures/thread.py", line 58, in run
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: .snakemake/log/2022-10-24T132312.559320.snakemake.log

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the build_renewable_profiles rule in Snakefile around line 377 and the command that generates resources/profile_offwind-dc.nc. Trace its availabilitymatrix call into atlite's gis.py, especially shape_availability around line 463, and reproduce the manually built europe-2013-era5 case. Done means the Snakemake command completes and produces the offshore-wind profile without the broadcasting error.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.