derrickoswald / derrickoswald/CIMSpark

partition by TopologicalIsland

Open
#4 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Scala
Stars
19
Forks
1
PR merge metrics
No merged PRs in 30d

Description

A possibility exists that partitioning according to TopologicalIsland may be beneficial for many use-cases.

Most electric distribution consists of radial networks, where the network downstream of a supply transformer is an isolated network that can be modeled independently, albeit with a single attachment to the grid at the transformer. The network topology processor identifies these "islands of connectivity" and generates one TopologicalIsland object, which all contained Terminal and ConnectivityNode objects reference.

In some cases ganged/parallel transformers are used to supply higher power requirements, so the phrase "a supply transformer" in the above description should be phrased as "a group of supply transformers that share a common bus".

The term _Trafokreis_ (english: transformer circuit) is used to refer to these islands serviced by a transformer group.

Not all islands correspond to a Trafokreis. Notable exceptions are:

* middle and high voltage islands (voltage ranges higher than the 400V distribution network)
* fragments where a normally open switch isolates a set of equipment from the network
* public lighting isolated by a 400:230 volt transformer - which in reality is it's own trafokreis

For some analysis use-cases, these topological islands can be processed in parallel independent of each other. It may be possible to partition the CIM RDD classes, based on their topological island, so as to bring all relevant objects to the same machine for processing a priori.

So, for example, the CIM classes related to the island for transformer(s) X would be assigned to the Spark worker Y. The number of topological islands far exceeds the number of workers, so each worker would have several islands.

The `RDD.coalesce()` method takes an optional [PartiionCoalescer](http://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.rdd.PartitionCoalescer) which generates an array of `PartitionGroup`s, each of which has an array of `Partition`s and their preferred location (machine name). But there doesn't seem to be a way to use anything except a HashPartitioner or RangePartitioner with the default RDD, so this will probably involve creating a subclass of RDD with the desired partitioning infrastructure.

The `RDD.groupBy` method takes an optional [Partitioner](http://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.Partitioner) that would allow the creation of individual RDD for each partition, but it's unclear how this would work.

In any case, partitioning like this would require a mapping table between CIM rdf:ID and partition (Trafokreis) __**for all elements**__. This would include items not in the topology, such as assets, locations, etc. It also probably requires a partition0 to contain elements shared between islands such as voltages and power system resource types, etc.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.