cockroachdb / cockroachdb/cockroach
server: can't connect a SQL server running a non-release build of v23.1 against a master/v23.2 KV layer
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
**Describe the problem**
I am trying the following combination:
- KV layer running either dev or release build off `master`
- SQL layer running dev build (NOT release build) from `release-23.1`
The SQL server fails to start with the following:
```
W230804 19:13:22.383400 360 server/settingswatcher/settings_watcher.go:330 ⋮ [T2,nsql?,rangefeed=‹settings-watcher›] 46 failed to set cluster version: ‹cannot upgrade to 1000023.1-16: node running 23.1›
[...]
I230804 19:13:22.404013 111 upgrade/upgrademanager/manager.go:238 ⋮ [T2,nsql3] 62 the last permanent upgrade (v22.1-42) does not appear to have completed; attempting to run all upgrades
I230804 19:13:22.404423 111 upgrade/upgrademanager/manager.go:283 ⋮ [T2,nsql3] 63 running permanent upgrade for version 0.0-2
W230804 19:13:22.405144 111 upgrade/upgrademanager/manager.go:762 ⋮ [T2,nsql3] 64 failed to check if migration already completed: ‹migration-job-find-already-completed›: no healthy sql instances available for planning
```
If I manually modify `settingswatcher/settings_watcher.go:330` to disable the check there, then I run into another error:
```
ERROR: server startup failed: cockroach server exited with error: preventing SQL server from starting because its binary version is too low for the tenant active version: server binary version = 23.1, tenant active version = 1000023.1-16
HINT: use a tenant binary whose version is at least 1000023.1-16
```
The error persists no matter how I set the env var `COCKROACH_UPGRADE_TO_DEV_VERSION=true`.
Interestingly, **if I use a release build this error does not occur**.
This was not caught in the roachtest because the roachtest uses release builds.
**Expected behavior**
A mixed version 23.1/23.2 connection using development (non-release) builds should work.
Epic: CRDB-26691
Jira issue: CRDB-30362
Contributor guide
Assessment
This issue has not been assessed yet.