Foundationdb loses all the data after renaming datacenter_id and setting up regions
- Dominant language
- C++
- Stars
- 16.7k
- Forks
- 1.6k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 126
Description
Problem statement
-------------------------
If fdbserver datacenter_id differs from the one was when the database was created, setting up regions causes deleting all data from the foundationdb cluster.
Steps to reproduce
-------------------------
1. Set up a three-node Primary FDB cluster with ``datacenter_id = dummy`` parameter
2. Create the database with ``configure new ssd double```
3. Restart the all fdbserver processes in the cluster with ``datacenter_id = dc1`` parameter
4. Make a simple region description in the regions.json file
```
{
"regions": [
{"datacenters": [{"id": "dc1", "priority": 6}]},
{"datacenters": [{"id": "dc2", "priority": -1}]}
]
}
```
5. Configure regions in the database with fdbcli:
``fileconfigure FORCE regions.json``
The FORCE is necessary because there is no any fdbserver processes in dc2 yet
6. Create and start another three fdb nodes with ``datacenter_id = dc2`` parameter
7. Try to start replication from dc1 to dc2 with fdbcli:
``configure usable_regions:=2``
Expected result
---------------------
The database should remain available. The data should start replicating from dc1 to dc2
Actual result
-----------------
```
WARNING: Long delay (Ctrl-C to interrupt)
The database is unavailable; type `status' for more information.
```
Findings
-----------
After trying to change `usable_regions` all the storage files of all fdbservers in dc1 get deleted and do not more exist.
Despite restarting fdbserver processes with the new datacenter_id = dc1, the old datacenter_id is still present in the system space of the database and reserves the locality tag 0.
```
fdb> getrangekeys \xff/tagLocalityList \xff/tagLocalityList0
Range limited to 25 keys
`\xff/tagLocalityList/\x01\x03\x00\x00\x00dummy'
fdb> get \xff/tagLocalityList/\x01\x03\x00\x00\x00dummy
`\xff/tagLocalityList/\x01\x03\x00\x00\x00dummy' is `\x01\x00\x01b\xb0\x00\xdb\x0f\x00'
```
The new datacenters_id s dynamically got the next locality tags 1 and 2 (recruitEverything->newTLogServers in fdbserver/masterserver.actor.cpp)
1. storageServerCore receives dbInfoChange
2. storageServerCore calls TagPartitionedLogSystem::peekSingle
3. TagPartitionedLogSystem::peekSingle calls TagPartitionedLogSystem::peekLocal
4. TagPartitionedLogSystem::peekLocal scans tLogs for a log with the locality 0 (got from the storage tag). But there are only logs with locality 1 and 2 and there are no logs with locality 0.
5. In this case TagPartitionedLogSystem::peekLocal traces event TLogPeekLocalNoBestSet and throws worker_removed()
6. the storageServerCore catches this exception worker_removed and calls storageServerTerminated
7. storageServerTerminated removes all data files, that causes the database unavailability.
Proposal
------------
- The best solution would be to automatically manage the locality dictionary in the system space and add the new datacenter_id with the same locality tag.
- Another acceptable solution is to add checks to configure and fileconfigure that all localities registered in \xff/tagLocalityList present in the region configuration
- Maybe add check on starting storageserver that it's datacenter_id is registered in the \xff/tagLocalityList and refuse to start with a clear error message
- The minimal is to mention in the documentation that changing datacenter_is is strongly forbidden after database has been created
Contributor guide
Research direction
Reproduce the sequence with fdbserver and fdbcli, including the datacenter_id change and regions.json configuration. Start in fdbserver/masterserver.actor.cpp, then trace TagPartitionedLogSystem::peekSingle, peekLocal, and the worker_removed path described in the report. Done means the database remains available and its storage files are not deleted when regions are configured.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100