cockroachdb / cockroachdb/cockroach

server: can't connect a SQL server running a non-release build of v23.1 against a master/v23.2 KV layer

Open
#108,207 1 comment 0 reactions 0 assignees View on GitHub
A-multitenancy C-bug T-db-server T-multitenant
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Describe the problem**

I am trying the following combination:
- KV layer running either dev or release build off `master`
- SQL layer running dev build (NOT release build) from `release-23.1`

The SQL server fails to start with the following:

```
W230804 19:13:22.383400 360 server/settingswatcher/settings_watcher.go:330 ⋮ [T2,nsql?,rangefeed=‹settings-watcher›] 46 failed to set cluster version: ‹cannot upgrade to 1000023.1-16: node running 23.1›
[...]
I230804 19:13:22.404013 111 upgrade/upgrademanager/manager.go:238 ⋮ [T2,nsql3] 62 the last permanent upgrade (v22.1-42) does not appear to have completed; attempting to run all upgrades
I230804 19:13:22.404423 111 upgrade/upgrademanager/manager.go:283 ⋮ [T2,nsql3] 63 running permanent upgrade for version 0.0-2
W230804 19:13:22.405144 111 upgrade/upgrademanager/manager.go:762 ⋮ [T2,nsql3] 64 failed to check if migration already completed: ‹migration-job-find-already-completed›: no healthy sql instances available for planning
```

If I manually modify `settingswatcher/settings_watcher.go:330` to disable the check there, then I run into another error:

```
ERROR: server startup failed: cockroach server exited with error: preventing SQL server from starting because its binary version is too low for the tenant active version: server binary version = 23.1, tenant active version = 1000023.1-16
HINT: use a tenant binary whose version is at least 1000023.1-16
```

The error persists no matter how I set the env var `COCKROACH_UPGRADE_TO_DEV_VERSION=true`.

Interestingly, **if I use a release build this error does not occur**.

This was not caught in the roachtest because the roachtest uses release builds.

**Expected behavior**

A mixed version 23.1/23.2 connection using development (non-release) builds should work.

Epic: CRDB-26691

Jira issue: CRDB-30362

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.