derrickoswald / derrickoswald/CIMSpark

Joining two VertexPartitions with different indexes is slow.

Open
#3 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Scala
Stars
19
Forks
1
PR merge metrics
No merged PRs in 30d

Description

In performing the network topology processing to create the TopologyNode and TopologyIsland RDD, numerous messages about _joining VertexPartitions with different indexes is slow_ are logged:

```
WARN impl.ShippableVertexPartitionOps: Joining two VertexPartitions with different indexes is slow.
```

It is unclear which operations are causing these messages or if they can be avoided. Google searches indicate that the VertexRDD that are used should be cached, but preliminary attempts to cache generated VertexRDD had no effect. Other processing which may be at fault is to update the ACDCTerminal, and ConnectivityNode RDD including their superclasses.

An attempt should be made to understand the origin of these messages and ameliorate them if possible.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.