apple / apple/foundationdb

Changing locality of processes

Open
#2,068 2 comments 0 reactions 0 assignees View on GitHub
documentation operations testing
Dominant language
C++
Stars
16.7k
Forks
1.6k
Avg merge
1d 20h
Merged PRs (30d)
126

Description

We managed to lose data on one of our test clusters by executing the following steps:

1. We had 5 machines running FDB 6.2.1 - running 12 procs each
1. Additionally we had one machine that ran an older version (the processes there couldn't join the cluster)
1. The cluster was configured with `configure new triple ssd` and afterwards we populated some generated test data (TPC-C)
1. We changed the config with `configure ssd three_data_hall` which caused the cluster to go down (this is expected as we didn't set the locality_data_hall)
1. We changed the config on all machines and set `locality_data_hall` so that two machines would always share the same value.
1. We restarted fdb on all machines through ansible.

After that, all data was gone.

This being said: I am not 100% convinced that this is the full story and I am not sure this was an fdb bug. However, it seems that changing localities on processes isn't super well tested in simulation. Because of that I think we should mention in the documentation that changing localities on processes is not supported.

Contributor guide

Open the contributing guide

Research direction

No file or test is identified in the issue. Review the documentation covering `configure`, `locality_data_hall`, and process localities, then verify the supported behavior with maintainers or existing tests; done means clearly documenting whether changing process localities is supported and any required procedure or warning.

Written by the indexing model from the issue text.

Assessment

Domain
databases, distributed-systems, documentation
Issue type
Documentation
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.