apple / apple/foundationdb

Recruitment should understand latency implications of data_hall

Open
#5,123 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
16.7k
Forks
1.6k
Avg merge
1d 20h
Merged PRs (30d)
126

Description

I have a 6 machine, 3 data hall cluster, running in three_data_hall, on 6.2.32. The RTT within a data_hall is 0.2ms, and across data_halls is 1ms. It recruited itself as follows:

```
{"data_hall":"DH1","processes":{"role":"master","count":1}}

{"data_hall":"DH2","processes":{"role":"log","count":2}}
{"data_hall":"DH2","processes":{"role":"proxy","count":2}}

{"data_hall":"DH3","processes":{"role":"log","count":2}}
{"data_hall":"DH3","processes":{"role":"proxy","count":1}}
{"data_hall":"DH3","processes":{"role":"resolver","count":1}}
```

Which is a bit odd. The master ended up all on its own, which is going to purely be a latency penalty. I would have expected the master being in DH1 to mean that logs/proxies/resolvers would be preferentially recruited locally in DH1 as well, but the opposite appeared to have occurred.

So I'd consider master/proxy/resolver/log in one data hall and proxy/log in the other to be acceptable. Master/proxy/log in one AZ and resolver/proxy/log in the other would be ideal, as then the latency of a commit would be equal regardless of which AZ's proxy you randomly decide to connect to. (I'd rather have all commits take 3ms than 50% be 4ms and 50% be 2ms.)

Contributor guide

Open the contributing guide

Research direction

The issue names no files, tests, or entry points. Start by tracing recruitment behavior for the three_data_hall configuration and how data_hall placement is selected; done means the resulting role placement accounts for inter-hall latency and matches an agreed recruitment policy.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
databases, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.