apple / apple/foundationdb

Foundationdb loses all the data after renaming datacenter_id and setting up regions

Open
#4,089 9 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
16.7k
Forks
1.6k
Avg merge
1d 20h
Merged PRs (30d)
126

Description

Problem statement
-------------------------
If fdbserver datacenter_id differs from the one was when the database was created, setting up regions causes deleting all data from the foundationdb cluster.

Steps to reproduce
-------------------------

1. Set up a three-node Primary FDB cluster with ``datacenter_id = dummy`` parameter
2. Create the database with ``configure new ssd double```
3. Restart the all fdbserver processes in the cluster with ``datacenter_id = dc1`` parameter
4. Make a simple region description in the regions.json file
```
{
"regions": [
{"datacenters": [{"id": "dc1", "priority": 6}]},
{"datacenters": [{"id": "dc2", "priority": -1}]}
]
}
```
5. Configure regions in the database with fdbcli:
``fileconfigure FORCE regions.json``
The FORCE is necessary because there is no any fdbserver processes in dc2 yet
6. Create and start another three fdb nodes with ``datacenter_id = dc2`` parameter
7. Try to start replication from dc1 to dc2 with fdbcli:
``configure usable_regions:=2``

Expected result
---------------------
The database should remain available. The data should start replicating from dc1 to dc2

Actual result
-----------------
```
WARNING: Long delay (Ctrl-C to interrupt)

The database is unavailable; type `status' for more information.
```

Findings
-----------

After trying to change `usable_regions` all the storage files of all fdbservers in dc1 get deleted and do not more exist.

Despite restarting fdbserver processes with the new datacenter_id = dc1, the old datacenter_id is still present in the system space of the database and reserves the locality tag 0.

```
fdb> getrangekeys \xff/tagLocalityList \xff/tagLocalityList0

Range limited to 25 keys
`\xff/tagLocalityList/\x01\x03\x00\x00\x00dummy'

fdb> get \xff/tagLocalityList/\x01\x03\x00\x00\x00dummy
`\xff/tagLocalityList/\x01\x03\x00\x00\x00dummy' is `\x01\x00\x01b\xb0\x00\xdb\x0f\x00'
```

The new datacenters_id s dynamically got the next locality tags 1 and 2 (recruitEverything->newTLogServers in fdbserver/masterserver.actor.cpp)

1. storageServerCore receives dbInfoChange
2. storageServerCore calls TagPartitionedLogSystem::peekSingle
3. TagPartitionedLogSystem::peekSingle calls TagPartitionedLogSystem::peekLocal
4. TagPartitionedLogSystem::peekLocal scans tLogs for a log with the locality 0 (got from the storage tag). But there are only logs with locality 1 and 2 and there are no logs with locality 0.
5. In this case TagPartitionedLogSystem::peekLocal traces event TLogPeekLocalNoBestSet and throws worker_removed()
6. the storageServerCore catches this exception worker_removed and calls storageServerTerminated
7. storageServerTerminated removes all data files, that causes the database unavailability.

Proposal
------------

- The best solution would be to automatically manage the locality dictionary in the system space and add the new datacenter_id with the same locality tag.
- Another acceptable solution is to add checks to configure and fileconfigure that all localities registered in \xff/tagLocalityList present in the region configuration
- Maybe add check on starting storageserver that it's datacenter_id is registered in the \xff/tagLocalityList and refuse to start with a clear error message
- The minimal is to mention in the documentation that changing datacenter_is is strongly forbidden after database has been created

Contributor guide

Open the contributing guide

Research direction

Reproduce the sequence with fdbserver and fdbcli, including the datacenter_id change and regions.json configuration. Start in fdbserver/masterserver.actor.cpp, then trace TagPartitionedLogSystem::peekSingle, peekLocal, and the worker_removed path described in the report. Done means the database remains available and its storage files are not deleted when regions are configured.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.