Edge case clustering issues

Version Checks (indicate both or one)

• This bug exists on the master branch of PyPSA-USA.

• This bug exists on the develop branch of PyPSA-USA.

The Issue

I am getting a clustering error on the simplification step when running Wisconsin and Ohio independently at reeds_zone resolution. This is quite similar to the issue discussed in Issue #646.

Steps To Reproduce

Run this config:

# docs :
scenario:
  interconnect: [eastern] #"usa|texas|western|eastern"
  clusters: [7a] # or 7, 7m, 6, 6a, 6m
  simpl: [70]
  opts: [3h]
  ll: [v1.0]
  sector: "" # G
  planning_horizons: [2030] 
foresight: 'perfect' # myopic, perfect

model_topology:
  transmission_network: 'reeds' # [reeds, tamu]
  topological_boundaries: 'reeds_zone' # [county, reeds_zone, state]
  interface_transmission_limits: false
  include: # mixed zone types not supported
    # reeds_zone: []
    reeds_state: ['WI'] # or "OH"
    # reeds_ba: []
  aggregate: # eligible keys: [reeds_zone, trans_reg]
    # trans_grp: []
    # reeds_zone: []

Error Message

Traceback (most recent call last):
  File "/home/trevor/miniforge3/envs/pypsa-usa/lib/python3.11/site-packages/pandas/core/indexes/base.py", line 3805, in get_loc
    return self._engine.get_loc(casted_key)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "index.pyx", line 167, in pandas._libs.index.IndexEngine.get_loc
  File "index.pyx", line 196, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/hashtable_class_helper.pxi", line 7081, in pandas._libs.hashtable.PyObjectHashTable.get_item
  File "pandas/_libs/hashtable_class_helper.pxi", line 7089, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: '1'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "index.pyx", line 768, in pandas._libs.index.BaseMultiIndexCodesEngine.get_loc
  File "/home/trevor/miniforge3/envs/pypsa-usa/lib/python3.11/site-packages/pandas/core/indexes/base.py", line 3812, in get_loc
    raise KeyError(key) from err
KeyError: '1'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/trevor/miniforge3/envs/pypsa-usa/lib/python3.11/site-packages/pandas/core/indexes/multi.py", line 3053, in get_loc
    return self._engine.get_loc(key)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "index.pyx", line 771, in pandas._libs.index.BaseMultiIndexCodesEngine.get_loc
KeyError: ('p75', '1')

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/trevor/miniforge3/envs/pypsa-usa/lib/python3.11/site-packages/pandas/core/series.py", line 1132, in __getitem__
    result = self._get_value(key)
             ^^^^^^^^^^^^^^^^^^^^
  File "/home/trevor/miniforge3/envs/pypsa-usa/lib/python3.11/site-packages/pandas/core/series.py", line 1237, in _get_value
    loc = self.index.get_loc(label)
          ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/trevor/miniforge3/envs/pypsa-usa/lib/python3.11/site-packages/pandas/core/indexes/multi.py", line 3055, in get_loc
    raise KeyError(key) from err
KeyError: ('p75', '1')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/trevor/miniforge3/envs/pypsa-usa/lib/python3.11/site-packages/pandas/core/indexes/base.py", line 3805, in get_loc
    return self._engine.get_loc(casted_key)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "index.pyx", line 167, in pandas._libs.index.IndexEngine.get_loc
  File "index.pyx", line 196, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/hashtable_class_helper.pxi", line 7081, in pandas._libs.hashtable.PyObjectHashTable.get_item
  File "pandas/_libs/hashtable_class_helper.pxi", line 7089, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: '1'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "index.pyx", line 768, in pandas._libs.index.BaseMultiIndexCodesEngine.get_loc
  File "/home/trevor/miniforge3/envs/pypsa-usa/lib/python3.11/site-packages/pandas/core/indexes/base.py", line 3812, in get_loc
    raise KeyError(key) from err
KeyError: '1'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/trevor/miniforge3/envs/pypsa-usa/lib/python3.11/site-packages/pandas/core/indexes/multi.py", line 3220, in _get_loc_level
    return (self._engine.get_loc(key), None)
            ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "index.pyx", line 771, in pandas._libs.index.BaseMultiIndexCodesEngine.get_loc
KeyError: ('p75', '1')

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/trevor/master/pypsa-usa/workflow/.snakemake/scripts/tmpbb5f37nj.simplify_network.py", line 292, in <module>
    clustering = clustering_for_n_clusters(
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/trevor/master/pypsa-usa/workflow/rules/../scripts/cluster_network.py", line 334, in clustering_for_n_clusters
    busmap = busmap_for_n_clusters(
             ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/trevor/master/pypsa-usa/workflow/rules/../scripts/cluster_network.py", line 314, in busmap_for_n_clusters
    .apply(busmap_for_country, include_groups=False)
     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/trevor/miniforge3/envs/pypsa-usa/lib/python3.11/site-packages/pandas/core/groupby/groupby.py", line 1819, in apply
    return self._python_apply_general(f, self._obj_with_exclusions)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/trevor/miniforge3/envs/pypsa-usa/lib/python3.11/site-packages/pandas/core/groupby/groupby.py", line 1885, in _python_apply_general
    values, mutated = self._grouper.apply_groupwise(f, data, self.axis)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/trevor/miniforge3/envs/pypsa-usa/lib/python3.11/site-packages/pandas/core/groupby/ops.py", line 919, in apply_groupwise
    res = f(group)
          ^^^^^^^^
  File "/home/trevor/master/pypsa-usa/workflow/rules/../scripts/cluster_network.py", line 290, in busmap_for_country
    n_clusters_per_region[x.name],
    ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^
  File "/home/trevor/miniforge3/envs/pypsa-usa/lib/python3.11/site-packages/pandas/core/series.py", line 1142, in __getitem__
    return self._get_values_tuple(key)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/trevor/miniforge3/envs/pypsa-usa/lib/python3.11/site-packages/pandas/core/series.py", line 1210, in _get_values_tuple
    indexer, new_index = self.index.get_loc_level(key)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/trevor/miniforge3/envs/pypsa-usa/lib/python3.11/site-packages/pandas/core/indexes/multi.py", line 3150, in get_loc_level
    loc, mi = self._get_loc_level(key, level=level)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/trevor/miniforge3/envs/pypsa-usa/lib/python3.11/site-packages/pandas/core/indexes/multi.py", line 3222, in _get_loc_level
    raise KeyError(key) from err
KeyError: ('p75', '1')

Anything else?

The best I can tell is it is an issue with the subnetworks be totally empty. For example, in WI, the subnetwork ("p75", "1") does not exist in the data structure.

When I inspect it, the sub_network, it consists of two buses (25682 and 20889 in this case)

However, these buses are only connected to each other, and have generators, but no loads attached to them.

My best guess is since this is an isolated sub_network with no load, something odd is happening in the busmap clustering? But I don't know the clustering details well enough off the top of my head to make a guess more insightful then that haha.

Start with workflow/scripts/cluster_network.py, especially busmap_for_country and busmap_for_n_clusters, then inspect the simplify_network.py call site shown in the traceback. Reproduce the supplied Wisconsin and Ohio reeds_zone configuration and trace the missing ('p75', '1') entry. Done means simplification completes without a KeyError for the isolated subnetwork case.