JanusGraph / JanusGraph/janusgraph

Proposal: Implement Distributed Balanced Partitioning via Linear Embedding (DBP-LE)

Open
#4,875 0 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Java
Stars
5.8k
Forks
1.2k
Avg merge
13h 53m
Merged PRs (30d)
6

Description

**Describe the feature:**
This proposal suggests implementing the **Distributed Balanced Partitioning via Linear Embedding (DBP-LE)** algorithm — originally developed by Aydin, Bateni & Mirrokni (Google Research, WSDM 2016) — as an optional graph partitioning strategy in JanusGraph.

DBP-LE addresses the challenge of *balanced partitioning* in large distributed graphs by:
1. Embedding vertices into a one-dimensional linear order using affinity (common-neighbor similarity).
2. Performing local refinements via *Minimum Linear Arrangement (MinLA)* and *RankSwap*.
3. Applying imbalance-aware postprocessing (sliding-window min-cut / DP optimization).

The output is a vertex ordering and balanced partition boundaries that minimize edge cuts between partitions.

This algorithm is highly scalable (MapReduce-friendly), easy to integrate with JanusGraph’s backend partitioner, and has been validated on real-world production graphs (Google Maps, Twitter, Friendster).

---

**Describe a specific use case for the feature:**
Distributed graph deployments in JanusGraph currently rely on simple hash or range partitioning, or external partitioners such as METIS.
These methods either:
- don’t scale to billions of edges, or
- produce suboptimal cut quality (leading to excessive cross-partition traversals).

Integrating DBP-LE as a native partitioning strategy would:

* Improve query performance and reduce network I/O by minimizing cross-machine edges.
* Provide balanced data distribution across storage backends.
* Enable better performance for iterative analytics (e.g., PageRank, Connected Components, community detection).
* Support machine-learning workloads (e.g., GNN training) that require balanced graph mini-batches with minimal boundary communication.

Example configuration:
```yaml
storage.partition.strategy: dbp_linear_embedding
storage.partition.alpha: 0.03
storage.partition.parts: 16

Additional context / references:

Paper: Distributed Balanced Partitioning via Linear Embedding — Aydin, Bateni, Mirrokni, WSDM 2016. DOI: [10.1145/2835776.2835829](https://doi.org/10.1145/2835776.2835829)

Expected benefits:

15–25 % reduction in edge cut size vs. METIS/FENNEL.

40 % fewer cross-shard queries in Google Maps case study.

Linear scalability to hundreds of millions of vertices.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing JanusGraph's backend partitioner and the cited DBP-LE paper, then compare the proposal with existing hash, range, and external partitioning approaches. The work is done when DBP-LE is integrated as an optional strategy with the proposed configuration and produces balanced partitions with reduced edge cuts.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.