cockroachdb / cockroachdb/cockroach

showing mix version when there is older node decommissioned

Open
#112,570 0 comments 0 reactions 0 assignees View on GitHub
A-check-on-console C-bug T-observability
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

When upgrading a cluster from an older version such as 20.2 with nodes decommissioned, the message of mixed version still shows.

Steps to repro:

Starting from lower version v20.2.4:
```
mohan@crlMBP-WW2265VQQRMTkw crdb-script % cockroach node status --decommission --insecure --host=localhost:26280
id | address | sql_address | build | started_at | updated_at | locality | is_available | is_live | gossiped_replicas | is_decommissioning | membership | is_draining
-----+-----------------+-----------------+---------+----------------------------------+----------------------------------+----------+--------------+---------+-------------------+--------------------+------------+--------------
1 | localhost:26280 | localhost:26280 | v20.2.4 | 2023-10-05 19:29:29.808678+00:00 | 2023-10-05 19:33:59.903856+00:00 | | true | true | 26 | false | active | false
2 | localhost:26281 | localhost:26281 | v20.2.4 | 2023-10-05 19:29:30.108539+00:00 | 2023-10-05 19:34:00.277391+00:00 | | true | true | 30 | false | active | false
3 | localhost:26285 | localhost:26285 | v20.2.4 | 2023-10-05 19:29:30.439375+00:00 | 2023-10-05 19:34:00.61433+00:00 | | true | true | 30 | false | active | false
4 | localhost:26282 | localhost:26282 | v20.2.4 | 2023-10-05 19:29:30.683427+00:00 | 2023-10-05 19:34:00.86296+00:00 | | true | true | 30 | false | active | false
5 | localhost:26284 | localhost:26284 | v20.2.4 | 2023-10-05 19:29:30.916892+00:00 | 2023-10-05 19:34:01.083776+00:00 | | true | true | 30 | false | active | false
6 | localhost:26283 | localhost:26283 | v20.2.4 | 2023-10-05 19:29:31.274567+00:00 | 2023-10-05 19:34:01.442809+00:00 | | true | true | 30 | false | active | false
(6 rows)
```

Decommissioned couple of nodes:
```
mohan@crlMBP-WW2265VQQRMTkw crdb-script % cockroach node status --decommission --insecure --host=localhost:26280
id | address | sql_address | build | started_at | updated_at | locality | is_available | is_live | gossiped_replicas | is_decommissioning | membership | is_draining
-----+-----------------+-----------------+---------+----------------------------------+----------------------------------+----------+--------------+---------+-------------------+--------------------+----------------+--------------
1 | localhost:26280 | localhost:26280 | v20.2.4 | 2023-10-05 19:29:29.808678+00:00 | 2023-10-05 19:42:50.923446+00:00 | | true | true | 36 | false | active | false
2 | localhost:26281 | localhost:26281 | v20.2.4 | 2023-10-05 19:29:30.108539+00:00 | 2023-10-05 19:42:51.273953+00:00 | | true | true | 36 | false | active | false
3 | localhost:26285 | localhost:26285 | v20.2.4 | 2023-10-05 19:29:30.439375+00:00 | 2023-10-05 19:42:51.593395+00:00 | | true | true | 36 | false | active | false
4 | localhost:26282 | localhost:26282 | v20.2.4 | 2023-10-05 19:29:30.683427+00:00 | 2023-10-05 19:42:51.846638+00:00 | | true | true | 0 | true | decommissioned | false
5 | localhost:26284 | localhost:26284 | v20.2.4 | 2023-10-05 19:29:30.916892+00:00 | 2023-10-05 19:42:52.102478+00:00 | | true | true | 0 | true | decommissioned | false
6 | localhost:26283 | localhost:26283 | v20.2.4 | 2023-10-05 19:29:31.274567+00:00 | 2023-10-05 19:42:52.439569+00:00 | | true | true | 0 | true | decommissioned | false
(6 rows)
```

Performing upgrade from v20.2.4 to v20.2.19:
```
mohan@crlMBP-WW2265VQQRMTkw crdb-script % cockroach node status --decommission --insecure --host=localhost:26280
id | address | sql_address | build | started_at | updated_at | locality | is_available | is_live | gossiped_replicas | is_decommissioning | membership | is_draining
-----+-----------------+-----------------+----------+----------------------------------+----------------------------------+----------+--------------+---------+-------------------+--------------------+----------------+--------------
1 | localhost:26280 | localhost:26280 | v20.2.19 | 2023-10-05 19:53:17.503502+00:00 | 2023-10-05 19:57:43.145433+00:00 | | true | true | 36 | false | active | false
2 | localhost:26281 | localhost:26281 | v20.2.19 | 2023-10-05 19:53:58.551529+00:00 | 2023-10-05 19:57:43.695757+00:00 | | true | true | 36 | false | active | false
3 | localhost:26285 | localhost:26285 | v20.2.19 | 2023-10-05 19:55:15.501127+00:00 | 2023-10-05 19:57:44.1305+00:00 | | true | true | 36 | false | active | false
4 | localhost:26282 | localhost:26282 | v20.2.4 | 2023-10-05 19:29:30.683427+00:00 | 2023-10-05 19:57:42.861739+00:00 | | true | true | 0 | true | decommissioned | false
5 | localhost:26284 | localhost:26284 | v20.2.4 | 2023-10-05 19:29:30.916892+00:00 | 2023-10-05 19:57:43.124821+00:00 | | true | true | 0 | true | decommissioned | false
6 | localhost:26283 | localhost:26283 | v20.2.4 | 2023-10-05 19:29:31.274567+00:00 | 2023-10-05 19:57:43.447178+00:00 | | true | true | 0 | true | decommissioned | false
(6 rows)
```

From v20.2.19 to v21.1.21:
```
mohan@crlMBP-WW2265VQQRMTkw crdb-script % cockroach node status --decommission --insecure --host=localhost:26280
id | address | sql_address | build | started_at | updated_at | locality | is_available | is_live | gossiped_replicas | is_decommissioning | membership | is_draining
-----+-----------------+-----------------+----------+----------------------------+----------------------------+----------+--------------+---------+-------------------+--------------------+----------------+--------------
1 | localhost:26280 | localhost:26280 | v21.1.21 | 2023-10-05 20:04:57.286154 | 2023-10-05 20:05:24.386756 | | true | true | 36 | false | active | false
2 | localhost:26281 | localhost:26281 | v21.1.21 | 2023-10-05 20:03:52.769039 | 2023-10-05 20:05:22.905458 | | true | true | 36 | false | active | false
3 | localhost:26285 | localhost:26285 | v21.1.21 | 2023-10-05 20:03:06.242069 | 2023-10-05 20:05:21.34317 | | true | true | 36 | false | active | false
4 | localhost:26282 | localhost:26282 | v20.2.4 | 2023-10-05 19:29:30.683427 | 2023-10-05 20:05:21.850807 | | true | true | 0 | true | decommissioned | false
5 | localhost:26284 | localhost:26284 | v20.2.4 | 2023-10-05 19:29:30.916892 | 2023-10-05 20:05:22.094606 | | true | true | 0 | true | decommissioned | false
6 | localhost:26283 | localhost:26283 | v20.2.4 | 2023-10-05 19:29:31.274567 | 2023-10-05 20:05:22.445398 | | true | true | 0 | true | decommissioned | false
(6 rows)
```

From v21.1.21 to v21.2.17:

```
mohan@crlMBP-WW2265VQQRMTkw crdb-script % cockroach node status --decommission --insecure --host=localhost:26280
id | address | sql_address | build | started_at | updated_at | locality | is_available | is_live | gossiped_replicas | is_decommissioning | membership | is_draining
-----+-----------------+-----------------+----------+----------------------------+----------------------------+----------+--------------+---------+-------------------+--------------------+----------------+--------------
1 | localhost:26280 | localhost:26280 | v21.2.17 | 2023-10-05 20:28:45.324503 | 2023-10-05 20:29:30.453077 | | true | true | 39 | false | active | false
2 | localhost:26281 | localhost:26281 | v21.2.17 | 2023-10-05 20:28:06.202638 | 2023-10-05 20:29:31.813404 | | true | true | 39 | false | active | false
3 | localhost:26285 | localhost:26285 | v21.2.17 | 2023-10-05 20:29:20.369879 | 2023-10-05 20:29:29.48055 | | true | true | 39 | false | active | false
4 | localhost:26282 | localhost:26282 | v20.2.4 | 2023-10-05 19:29:30.683427 | 2023-10-05 20:28:54.396645 | | false | false | 0 | true | decommissioned | false
5 | localhost:26284 | localhost:26284 | v20.2.4 | 2023-10-05 19:29:30.916892 | 2023-10-05 20:28:54.396649 | | false | false | 0 | true | decommissioned | false
6 | localhost:26283 | localhost:26283 | v20.2.4 | 2023-10-05 19:29:31.274567 | 2023-10-05 20:28:54.396694 | | false | false | 0 | true | decommissioned | false
(6 rows)
```

From v21.2.17 to v22.1.22:
```
mohan@crlMBP-WW2265VQQRMTkw crdb-script % cockroach node status --decommission --insecure --host=localhost:26280
id | address | sql_address | build | started_at | updated_at | locality | is_available | is_live | gossiped_replicas | is_decommissioning | membership | is_draining
-----+-----------------+-----------------+----------+----------------------------+----------------------------+----------+--------------+---------+-------------------+--------------------+----------------+--------------
1 | localhost:26280 | localhost:26280 | v22.1.22 | 2023-10-05 20:36:05.687704 | 2023-10-05 20:36:37.314762 | | true | true | 50 | false | active | false
2 | localhost:26281 | localhost:26281 | v22.1.22 | 2023-10-05 20:36:34.627462 | 2023-10-05 20:36:39.251015 | | true | true | 50 | false | active | false
3 | localhost:26285 | localhost:26285 | v22.1.22 | 2023-10-05 20:35:38.573009 | 2023-10-05 20:36:37.213812 | | true | true | 50 | false | active | false
4 | localhost:26282 | localhost:26282 | v20.2.4 | 2023-10-05 19:29:30.683427 | 2023-10-05 20:35:46.799433 | | false | false | 0 | true | decommissioned | false
5 | localhost:26284 | localhost:26284 | v20.2.4 | 2023-10-05 19:29:30.916892 | 2023-10-05 20:35:46.79944 | | false | false | 0 | true | decommissioned | false
6 | localhost:26283 | localhost:26283 | v20.2.4 | 2023-10-05 19:29:31.274567 | 2023-10-05 20:35:46.799445 | | false | false | 0 | true | decommissioned | false
(6 rows)
```

From v22.1.22 to v22.2.14:
```
mohan@crlMBP-WW2265VQQRMTkw crdb-script % cockroach node status --decommission --insecure --host=localhost:26280
id | address | sql_address | build | started_at | updated_at | locality | is_available | is_live | gossiped_replicas | is_decommissioning | membership | is_draining
-----+-----------------+-----------------+----------+--------------------------------------+--------------------------------------+----------+--------------+---------+-------------------+--------------------+----------------+--------------
1 | localhost:26280 | localhost:26280 | v22.2.14 | 2023-10-05 20:47:27.016378 +0000 UTC | 2023-10-05 20:48:12.14236 +0000 UTC | | true | true | 55 | false | active | false
2 | localhost:26281 | localhost:26281 | v22.2.14 | 2023-10-05 20:47:04.90782 +0000 UTC | 2023-10-05 20:48:12.543417 +0000 UTC | | true | true | 55 | false | active | false
3 | localhost:26285 | localhost:26285 | v22.2.14 | 2023-10-05 20:48:04.061629 +0000 UTC | 2023-10-05 20:48:08.687634 +0000 UTC | | true | true | 55 | false | active | false
4 | localhost:26282 | localhost:26282 | v20.2.4 | 2023-10-05 19:29:30.683427 +0000 UTC | 2023-10-05 20:47:14.564687 +0000 UTC | | false | false | 0 | true | decommissioned | false
5 | localhost:26284 | localhost:26284 | v20.2.4 | 2023-10-05 19:29:30.916892 +0000 UTC | 2023-10-05 20:47:14.564689 +0000 UTC | | false | false | 0 | true | decommissioned | false
6 | localhost:26283 | localhost:26283 | v20.2.4 | 2023-10-05 19:29:31.274567 +0000 UTC | 2023-10-05 20:47:14.564692 +0000 UTC | | false | false | 0 | true | decommissioned | false
(6 rows)
```

From v22.2.14 to v23.1.11 latest:
```
mohan@crlMBP-WW2265VQQRMTkw crdb-script % cockroach node status --decommission --insecure --host=localhost:26280
id | address | sql_address | build | started_at | updated_at | locality | is_available | is_live | gossiped_replicas | is_decommissioning | membership | is_draining
-----+-----------------+-----------------+----------+--------------------------------------+--------------------------------------+----------+--------------+---------+-------------------+--------------------+----------------+--------------
1 | localhost:26280 | localhost:26280 | v23.1.11 | 2023-10-05 20:52:41.439259 +0000 UTC | 2023-10-05 20:53:35.79657 +0000 UTC | | true | true | 63 | false | active | false
2 | localhost:26281 | localhost:26281 | v23.1.11 | 2023-10-05 20:53:30.29424 +0000 UTC | 2023-10-05 20:53:36.675109 +0000 UTC | | true | true | 61 | false | active | false
3 | localhost:26285 | localhost:26285 | v23.1.11 | 2023-10-05 20:52:22.171399 +0000 UTC | 2023-10-05 20:53:37.552117 +0000 UTC | | true | true | 61 | false | active | false
4 | localhost:26282 | localhost:26282 | v20.2.4 | 2023-10-05 19:29:30.683427 +0000 UTC | 2023-10-05 20:53:09.438299 +0000 UTC | | false | false | 0 | true | decommissioned | false
5 | localhost:26284 | localhost:26284 | v20.2.4 | 2023-10-05 19:29:30.916892 +0000 UTC | 2023-10-05 20:53:09.438397 +0000 UTC | | false | false | 0 | true | decommissioned | false
6 | localhost:26283 | localhost:26283 | v20.2.4 | 2023-10-05 19:29:31.274567 +0000 UTC | 2023-10-05 20:53:09.438411 +0000 UTC | | false | false | 0 | true | decommissioned | false
(6 rows)
```

On the UI:

Screenshot 2023-10-17 at 3 59 38 PM

Screenshot 2023-10-17 at 3 59 46 PM

Jira issue: CRDB-32497

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.