apple / apple/foundationdb

'Replication health' continues to be in the '(Re)initializing automatic data distribution' phase

Open
#9,163 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
16.7k
Forks
1.6k
Avg merge
1d 20h
Merged PRs (30d)
126

Description

FDB version 7.2.0
Cluster size: 18 nodes in total, Redundancy mode use three_datacenter, 6 nodes per DC

Here is what I see in fdbcli:
```
# fdbcli
Using cluster file `/etc/foundationdb/fdb.cluster'.

The database is available, but has issues (type 'status' for more information).

Welcome to the fdbcli. For help, type `help'.
fdb>
fdb>
fdb> status

Using cluster file `/etc/foundationdb/fdb.cluster'.

Configuration:
Redundancy mode - three_datacenter
Storage engine - ssd-2
Encryption at-rest - disabled
Coordinators - 7
Usable Regions - 1

Cluster:
FoundationDB processes - 502
Zones - 502
Machines - 18
Memory availability - 8.0 GB per process on machine with least available
Retransmissions rate - 78 Hz
Fault Tolerance - 3 zones
Server time - 01/16/23 10:34:52

Data:
Replication health - (Re)initializing automatic data distribution
Moving data - unknown (initializing)
Sum of key-value sizes - unknown
Disk space used - 50.634 GB

Operating space:
Storage server - 1697.8 GB free on most full server
Log server - 1679.8 GB free on most full server

Workload:
Read rate - 303 Hz
Write rate - 87 Hz
Transactions started - 104 Hz
Transactions committed - 87 Hz
Conflict rate - 0 Hz

Backup and DR:
Running backups - 0
Running DRs - 0

Client time: 01/16/23 10:34:52

fdb>

```

The data_distributor process log has the following error:
```

# addr2line -e fdbserver.debug -p -C -f -i 0x7faca699d630 0x4680045 0x1794715 0x174c456 0x1765979 0x176b7ac 0x17c6706 0x17c6b73 0x17c6c5b 0x17ce0e5 0x15a3040 0x438bb32 0xc82397 0x7faca65e2555 0xd01412
?? ??:0
free_fastpath at /home/foundationdb_ci/foundationdb_build_output/dbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdb/Jemalloc_project-prefix/src/Jemalloc_project/src/jemalloc.c:3085
(inlined by) free at /home/foundationdb_ci/foundationdb_build_output/dbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdb/Jemalloc_project-prefix/src/Jemalloc_project/src/jemalloc.c:3161
__gnu_cxx::new_allocator >::deallocate(std::_Rb_tree_node*, unsigned long) at /opt/rh/devtoolset-8/root/usr/include/c++/8/ext/new_allocator.h:125
(inlined by) std::allocator_traits > >::deallocate(std::allocator >&, std::_Rb_tree_node*, unsigned long) at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/alloc_traits.h:462
(inlined by) std::_Rb_tree, std::less, std::allocator >::_M_put_node(std::_Rb_tree_node*) at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/stl_tree.h:603
(inlined by) std::_Rb_tree, std::less, std::allocator >::_M_drop_node(std::_Rb_tree_node*) at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/stl_tree.h:670
(inlined by) std::_Rb_tree, std::less, std::allocator >::_M_erase(std::_Rb_tree_node*) at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/stl_tree.h:1874
std::string::_M_rep() const at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/basic_string.h:3322
(inlined by) std::basic_string, std::allocator >::~basic_string() at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/basic_string.h:3640
(inlined by) void __gnu_cxx::new_allocator >::destroy(std::string*) at /opt/rh/devtoolset-8/root/usr/include/c++/8/ext/new_allocator.h:140
(inlined by) void std::allocator_traits > >::destroy(std::allocator >&, std::string*) at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/alloc_traits.h:487
(inlined by) std::_Rb_tree, std::less, std::allocator >::_M_destroy_node(std::_Rb_tree_node*) at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/stl_tree.h:661
(inlined by) std::_Rb_tree, std::less, std::allocator >::_M_drop_node(std::_Rb_tree_node*) at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/stl_tree.h:669
(inlined by) std::_Rb_tree, std::less, std::allocator >::_M_erase(std::_Rb_tree_node*) at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/stl_tree.h:1874
(inlined by) std::_Rb_tree, std::less, std::allocator >::~_Rb_tree() at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/stl_tree.h:965
(inlined by) std::set, std::allocator >::~set() at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/stl_set.h:281
(inlined by) DDTeamCollection::isValidLocality(Reference, LocalityData const&) const at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/fdbserver/DDTeamCollection.actor.cpp:3696
Reference::~Reference() at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/include/flow/FastRef.h:125 (discriminator 1)
(inlined by) DDTeamCollection::addBestMachineTeams(int) at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/fdbserver/DDTeamCollection.actor.cpp:4068 (discriminator 1)
DDTeamCollection::addTeamsBestOf(int, int, int) at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/fdbserver/DDTeamCollection.actor.cpp:4479
DDTeamCollectionImpl::BuildTeamsActorState::a_body1cont2(int) at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/fdbserver/DDTeamCollection.actor.cpp:599
DDTeamCollectionImpl::BuildTeamsActorState::a_body1cont1break1(int) at /home/foundationdb_ci/foundationdb_build_output/dbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdb/fdbserver/DDTeamCollection.actor.g.cpp:3618
DDTeamCollectionImpl::BuildTeamsActorState::a_body1cont1loopBody1(int) at /home/foundationdb_ci/foundationdb_build_output/dbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdb/fdbserver/DDTeamCollection.actor.g.cpp:3605 (discriminator 5)
DDTeamCollectionImpl::BuildTeamsActorState::a_body1cont1loopHead1(int) at /home/foundationdb_ci/foundationdb_build_output/dbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdb/fdbserver/DDTeamCollection.actor.g.cpp:3574
(inlined by) DDTeamCollectionImpl::BuildTeamsActorState::a_body1cont1(Void const&, int) at /home/foundationdb_ci/foundationdb_build_output/dbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdb/fdbserver/DDTeamCollection.actor.g.cpp:3365
(inlined by) DDTeamCollectionImpl::BuildTeamsActorState::a_body1when1(Void const&, int) at /home/foundationdb_ci/foundationdb_build_output/dbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdb/fdbserver/DDTeamCollection.actor.g.cpp:3380
(inlined by) DDTeamCollectionImpl::BuildTeamsActorState::a_callback_fire(ActorCallback*, Void const&) at /home/foundationdb_ci/foundationdb_build_output/dbdbdbdbdbdbdbdbdbdbdbdbdbdbdbdb/fdbserver/DDTeamCollection.actor.g.cpp:3401
(inlined by) ActorCallback::fire(Void const&) at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/include/flow/flow.h:1313
void SAV::send(Void&&) at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/include/flow/flow.h:654
Promise::~Promise() at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/include/flow/flow.h:922
(inlined by) N2::Net2::PromiseTask::~PromiseTask() at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/Net2.actor.cpp:253
(inlined by) N2::Net2::PromiseTask::operator()() at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/Net2.actor.cpp:260
(inlined by) N2::Net2::run() at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/flow/Net2.actor.cpp:1492
main at /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/fdbserver/fdbserver.actor.cpp:2310 (discriminator 4)
?? ??:0
_start at ??:?

```

![image](https://user-images.githubusercontent.com/59597542/212674110-decbc636-4757-46ec-9587-6e70b0ccf59c.png)

What could be the reason?

Contributor guide

Open the contributing guide

Research direction

Start with the fdbcli status output and the data_distributor trace, then inspect DDTeamCollection.actor.cpp at isValidLocality and the BuildTeamsActor call sites named in the stack trace. Reproduce or trace the automatic data-distribution initialization behavior and determine whether the reported failure explains the persistent status; done means identifying a supported cause and a concrete corrective direction.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.