Recruitment should understand latency implications of data_hall
- Dominant language
- C++
- Stars
- 16.7k
- Forks
- 1.6k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 126
Description
I have a 6 machine, 3 data hall cluster, running in three_data_hall, on 6.2.32. The RTT within a data_hall is 0.2ms, and across data_halls is 1ms. It recruited itself as follows:
```
{"data_hall":"DH1","processes":{"role":"master","count":1}}
{"data_hall":"DH2","processes":{"role":"log","count":2}}
{"data_hall":"DH2","processes":{"role":"proxy","count":2}}
{"data_hall":"DH3","processes":{"role":"log","count":2}}
{"data_hall":"DH3","processes":{"role":"proxy","count":1}}
{"data_hall":"DH3","processes":{"role":"resolver","count":1}}
```
Which is a bit odd. The master ended up all on its own, which is going to purely be a latency penalty. I would have expected the master being in DH1 to mean that logs/proxies/resolvers would be preferentially recruited locally in DH1 as well, but the opposite appeared to have occurred.
So I'd consider master/proxy/resolver/log in one data hall and proxy/log in the other to be acceptable. Master/proxy/log in one AZ and resolver/proxy/log in the other would be ideal, as then the latency of a commit would be equal regardless of which AZ's proxy you randomly decide to connect to. (I'd rather have all commits take 3ms than 50% be 4ms and 50% be 2ms.)
Contributor guide
Research direction
The issue names no files, tests, or entry points. Start by tracing recruitment behavior for the three_data_hall configuration and how data_hall placement is selected; done means the resulting role placement accounts for inter-hall latency and matches an agreed recruitment policy.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- databases, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100