cockroachdb / cockroachdb/cockroach
Add a recommended, non-deprecated, way to check for critical localities.
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
**Is your feature request related to a problem? Please describe.**
CRDB currently has a good way to check for critical localities: https://www.cockroachlabs.com/docs/stable/query-replication-reports#find-out-which-databases-and-tables-have-ranges-in-critical-localities
Critical localities there are regions that if they were to become unavailable, would lead to a set of ranges becoming unavailable.
For example, if I have these localities arising from a 3 node cluster with 3x replication on all ranges/databases:
```
root@localhost:26257/defaultdb> show regions;
region | zones | database_names | primary_region_of | secondary_region_of
---------+----------------+----------------+-------------------+----------------------
123 | {blbla,blbla1} | {} | {} | {}
12345 | {zzzz} | {} | {} | {}
```
Then the output of my critical localities would be:
```
root@localhost:26257/defaultdb> SELECT * FROM system.replication_critical_localities WHERE at_risk_ranges > 0;
zone_id | subzone_id | locality | report_id | at_risk_ranges
----------+------------+------------+-----------+-----------------
0 | 0 | region=123 | 2 | 2
1 | 0 | region=123 | 2 | 57
16 | 0 | region=123 | 2 | 1
17 | 0 | region=123 | 2 | 2
22 | 0 | region=123 | 2 | 1
(5 rows)
```
This works great, and alerts me to which localities absolutely cannot be taken down.
However, this has been deprecated, as described in a banner on [top of the linked page](https://www.cockroachlabs.com/docs/stable/query-replication-reports).
That deprecation notice now refers to the [critical nodes status endpoint](https://www.cockroachlabs.com/docs/v24.1/monitoring-and-alerting#critical-nodes-endpoint) for monitoring the clusters replication and data placement health.
That page even has an example titled [Replication status - ranges in critical localities](https://www.cockroachlabs.com/docs/v24.1/monitoring-and-alerting#replication-status-ranges-in-critical-localities).
That title makes it seem like exactly what I'm after, but reading through the example, and replicating it myself shows that it only reports on critical nodes, and not on critical localities. This difference is very important since a cluster may not have any critical nodes, while having critical localities, because all of a range's replicas may be living in the same locality.
Going back to the earlier example cluster, where the locality of `region=123` was critical, we see that the new critical nodes endpoint shows no problems. (Command taken verbatim from the "Replication status - ranges in critical localities" example)
```
$ curl -X POST http://localhost:8080/_status/critical_nodes
{
"criticalNodes": [
],
"report": {
"underReplicated": [
],
"overReplicated": [
],
"violatingConstraints": [
],
"unavailable": [
],
"unavailableNodeIds": [
]
}
}
```
In short, the new critical_nodes endpoint is inadequate as a replacement for system.replication_critical_localities, which seems like it's being deprecated.
**Describe the solution you'd like**
I would like a non-deprecated method of checking on critical localities, the behavior of which would preferably exactly match what `SELECT * FROM system.replication_critical_localities WHERE at_risk_ranges > 0;` gives me now.
**Describe alternatives you've considered**
- Keep using `SELECT * FROM system.replication_critical_localities WHERE at_risk_ranges > 0;`: This will apparently be wrong/removed in future releases
- Somehow query range information myself & write software to parse through that output to check if any localities are critical: Seems like a lot of work, and I'm not sure if there's a good way to do this at all, but if you deem that this is a good way forward, I'd likely pursue this.
**Additional context**
N/A
Jira issue: CRDB-40610
Contributor guide
Assessment
This issue has not been assessed yet.