7.1 with tenanting upgrade to 7.3 crash
- Dominant language
- C++
- Stars
- 16.7k
- Forks
- 1.6k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 126
Description
I’ve been testing 7.1 → 7.3 upgrades with tenanted data due to a discussion in [Tenants & golang - #8 by newhook](https://forums.foundationdb.org/t/tenants-golang/4408/8) which lead to [Tenant feature metadata changes in 7.2 release](https://forums.foundationdb.org/t/tenant-feature-metadata-changes-in-7-2-release/3459).
I've been testing:
foundationdb-clients_7.1.40-1_amd64.deb foundationdb-server_7.1.40-1_amd64.deb
upgrading to:
foundationdb-clients_7.3.36-1_amd64.deb foundationdb-server_7.3.36-1_amd64.deb
What I discovered was if I have tenanted data in the 7.1 database I end up crashing a 7.3 server on startup. I get one of two crashes:
```
Apr 10 18:25:02 staging-c-matthew-test fdbmonitor[60480]: LogGroup="default" Process="fdbserver.4500": FDBD joined cluster.
Apr 10 18:25:02 staging-c-matthew-test fdbmonitor[60480]: LogGroup="default" Process="fdbserver.4500": Assertion prefix.size() == TenantAPI::PREFIX_SIZE failed @ /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/fdbclient/Tenant.cpp 45:
Apr 10 18:25:02 staging-c-matthew-test fdbmonitor[60480]: LogGroup="default" Process="fdbserver.4500": addr2line -e fdbserver.debug -p -C -f -i 0x54a310c 0x4c3d11e 0x33f75ac 0x33f7388 0x33f68a4 0x33f5f43 0x33f5673 0x33f4caa 0x33f4683 0x33f3cc9 0x33f35bc 0x335840b 0x34279cb 0x342791d 0x34262c7 0x289b8f8 0x279bad8 0x279b985 0x279b859 0x279c28e 0x278bb08 0x5385828 0x5385663 0x5390c68 0x5390e68 0x5391ce8 0x5392148 0x5500a98 0x32b1a4b 0x7804a5429d90
```
or when I was running in memory mode, which I guess is occurring in log replay.
```
Apr 10 18:18:15 staging-c-matthew-test fdbmonitor[59481]: LogGroup="default" Process="fdbserver.4500": Assertion false failed @ /home/foundationdb_ci/src/oOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOoOo/foundationdb/fdbserver/storageserver.actor.cpp 10730:
Apr 10 18:18:15 staging-c-matthew-test fdbmonitor[59481]: LogGroup="default" Process="fdbserver.4500": addr2line -e fdbserver.debug -p -C -f -i 0x54a310c 0x34685ae 0x3466547 0x33d818e 0x33d62fb 0x33d57a9 0x33d546c 0x33d487a 0x33d3918 0x33d34da 0x33d17d4 0x29f4888 0x29f40c6 0x1ce5298 0x1ce4d03 0x29f1f08 0x29f2d78 0x29f292a 0x29eeb98 0x29ee31c 0x1e8cfc8 0x1e8cd0a 0x534de31 0x534d6d3 0x5500a98 0x32b1a4b 0x7db267e29d90
```
Are these expected? Anyway around them? Any advice?
Contributor guide
Research direction
Reproduce the 7.1.40-to-7.3.36 upgrade with tenanted data, then inspect the assertion locations in fdbclient/Tenant.cpp and fdbserver/storageserver.actor.cpp using the supplied crash traces. Done means determining whether the crash is expected and documenting or implementing a supported workaround or fix.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 42/100