apple / apple/foundationdb

7.1 with tenanting upgrade to 7.3 crash

Open
#11,292 4 comments 0 reactions 0 assignees View on GitHub
help wanted
Dominant language
C++
Stars
16.7k
Forks
1.6k
Avg merge
1d 20h
Merged PRs (30d)
126

Description

I’ve been testing 7.1 → 7.3 upgrades with tenanted data due to a discussion in [Tenants & golang - #8 by newhook](https://forums.foundationdb.org/t/tenants-golang/4408/8) which lead to [Tenant feature metadata changes in 7.2 release](https://forums.foundationdb.org/t/tenant-feature-metadata-changes-in-7-2-release/3459).

I've been testing:

foundationdb-clients_7.1.40-1_amd64.deb foundationdb-server_7.1.40-1_amd64.deb

upgrading to:

foundationdb-clients_7.3.36-1_amd64.deb foundationdb-server_7.3.36-1_amd64.deb

What I discovered was if I have tenanted data in the 7.1 database I end up crashing a 7.3 server on startup. I get one of two crashes:

```
Apr 10 18:25:02 staging-c-matthew-test fdbmonitor[60480]: LogGroup="default" Process="fdbserver.4500": FDBD joined cluster.
Apr 10 18:25:02 staging-c-matthew-test fdbmonitor[60480]: LogGroup="default" Process="fdbserver.4500": Assertion prefix.size() == TenantAPI::PREFIX_SIZE failed @ /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/fdbclient/Tenant.cpp 45:
Apr 10 18:25:02 staging-c-matthew-test fdbmonitor[60480]: LogGroup="default" Process="fdbserver.4500": addr2line -e fdbserver.debug -p -C -f -i 0x54a310c 0x4c3d11e 0x33f75ac 0x33f7388 0x33f68a4 0x33f5f43 0x33f5673 0x33f4caa 0x33f4683 0x33f3cc9 0x33f35bc 0x335840b 0x34279cb 0x342791d 0x34262c7 0x289b8f8 0x279bad8 0x279b985 0x279b859 0x279c28e 0x278bb08 0x5385828 0x5385663 0x5390c68 0x5390e68 0x5391ce8 0x5392148 0x5500a98 0x32b1a4b 0x7804a5429d90
```
or when I was running in memory mode, which I guess is occurring in log replay.
```
Apr 10 18:18:15 staging-c-matthew-test fdbmonitor[59481]: LogGroup="default" Process="fdbserver.4500": Assertion false failed @ /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/fdbserver/storageserver.actor.cpp 10730:
Apr 10 18:18:15 staging-c-matthew-test fdbmonitor[59481]: LogGroup="default" Process="fdbserver.4500": addr2line -e fdbserver.debug -p -C -f -i 0x54a310c 0x34685ae 0x3466547 0x33d818e 0x33d62fb 0x33d57a9 0x33d546c 0x33d487a 0x33d3918 0x33d34da 0x33d17d4 0x29f4888 0x29f40c6 0x1ce5298 0x1ce4d03 0x29f1f08 0x29f2d78 0x29f292a 0x29eeb98 0x29ee31c 0x1e8cfc8 0x1e8cd0a 0x534de31 0x534d6d3 0x5500a98 0x32b1a4b 0x7db267e29d90
```
Are these expected? Anyway around them? Any advice?

Contributor guide

Open the contributing guide

Research direction

Reproduce the 7.1.40-to-7.3.36 upgrade with tenanted data, then inspect the assertion locations in fdbclient/Tenant.cpp and fdbserver/storageserver.actor.cpp using the supplied crash traces. Done means determining whether the crash is expected and documenting or implementing a supported workaround or fix.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.