cockroachdb / cockroachdb/cockroach
server: tenant shutdown via `ALTER TENANT .. STOP SERVICE` is always ungraceful
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
This feels like a low priority paper-cut to fix, since this command isn't used widely. Here's a longer explanation:
It would be nice to strive to shut the tenant down gracefully for a couple of reasons. For example, if the system tenant quickly restarts the service, I noticed that jobs running before shutdown cannot be claimed until the dead sql instance's session has expired. If the tenant gracefully shutdown, i believe the restarted tenant could immediately resume the jobs.
To reproduce, run TestServerStartStop in serverccl, and look for:
```
E250919 17:50:00.772653 3307 1@server/drain.go:591 [n1,tenant-orchestration,tenant=hello,call-graceful-drain] 229 graceful drain failed: ba: Get [/Tenant/3/Table/46/2/"\x80"/1/0], [txn: 4863dc6e], [can-forward-ts] RPC error: grpc: operation not allowed when in service mode "stopping" [code 16/Unauthenticated]
I250919 17:50:00.772708 2370 server/server_controller_channel_orchestrator.go:442 [n1,tenant-orchestration,tenant=hello] 230 tenant "hello" finishing their own control loop
I250919 17:50:00.772740 2371 server/server_controller_channel_orchestrator.go:477 [n1,tenan
```
This likely regressed when we introduced the stopping state, to fix a much worse bug [here.](https://github.com/cockroachdb/cockroach/pull/117677).
The fix here is likely to add a new DRAINING state before the stopping state, where graceful shutdown could occur. That being said, tenant server shutdown (and startup) needs to be simplified. So, perhaps before taking this issue on, we should refactor this code.
Original slack thread here: https://cockroachlabs.slack.com/archives/C07T05244KZ/p1758304678161739
Jira issue: CRDB-54652
Contributor guide
Assessment
This issue has not been assessed yet.