apple / apple/foundationdb

Visualization for recruited roles in a cluster

Open
#3,074 2 comments 8 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
16.7k
Forks
1.6k
Avg merge
1d 20h
Merged PRs (30d)
126

Description

In digging into unhealthy clusters, I seem to spend a lot of time trying to figure out which pieces of the system are communicating with which other pieces of the system. Figuring out which Transaction log is the preferred location for which tag, or which log router is pulling from which TLog, or what generation a log router belongs to all slows down the debugging process. It'd be great to just have a way to get the IDs of recruited roles and arrows of how they're connected quickly.

It looks like if we had the raw information available, then graphviz lets us output a declarative description of the cluster, and it will handle the rendering. Subgraph clusters allow one to put a labelled box around a collection of nodes. HTML-like labels allow attaching tables of information to a process.

For example:

```
digraph FDB {
subgraph cluster_generation_1 {
label = "Generation 1";
subgraph cluster_DC1 {
label = "DC1"
Proxies [shape=plaintext
label=<
Proxy0:af9380
Proxy1:dc8947
Proxy2:eb8127
>]
Log0 [ shape=plaintext
label=<ID:aa827>]
}
subgraph cluster_DC2 {
label = "DC2"
subgraph cluster_SatelliteLogs {
label = "Satellite Logs"
SLog0 [shape=plaintext
label=<ID:192347>]
}
}
subgraph cluster_DC3 {
label = "DC3"
subgraph cluster_LogRouter {
label = "LogRouters"
LR0 [shape=plaintext
label=<ID:deof75>]
}
}
Proxies -> SLog0;
Proxies -> Log0;
SLog0 -> LR0;
}
}
```

becomes

image

However, the raw information doesn't seem to be readily available. Things related to transaction logs could probably be read from coordinated state, but other things, like the IDs of recruited log router instances and what generation they belong to, doesn't exist in an easy to access place right now.

Contributor guide

Open the contributing guide

Research direction

The issue names no files, tests, or entry points. Start by investigating coordinated state for transaction-log relationships and where recruited log-router IDs and generations are tracked. Done would mean exposing the relevant role IDs and connections in a form that can produce the proposed Graphviz cluster visualization.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
data-visualization, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.