cockroachdb / cockroachdb/cockroach
roachtest,multitenant: `cockroach node drain` do not work when run for first time
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
**Describe the problem**
While enabling `drain/early-exit-conn-wait` roachtest in [`drain.go`](https://github.com/cockroachdb/cockroach/blob/049c30ac6956fa8f307ba2adae8b4388f5907859/pkg/cmd/roachtest/tests/drain.go) for secondary tenants (starting with shared process) I observed that `health?ready=1` do not return 50X response code. On reproducing locally I confirmed that if we trigger `cockroach node drain` _without `shutdown`_ it will remain stuck after logging,
```
I241029 06:09:30.528178 2042 server/server_controller_channel_orchestrator.go:533 ⋮ [n1,tenant-orchestration,tenant=‹test›] 165 propagate-graceful-drain task terminating
I241029 06:09:30.532992 878 server/server_controller_channel_orchestrator.go:286 ⋮ [T1,Vsystem,n1] 166 propagate-close task terminating
```
The node status is still live. But if I run the drain command again, it succeeds.
**To Reproduce**
I'm consistently able to reproduce this by running the below command,
```sh
cockroach node drain --self --drain-wait=60s --insecure --host=localhost --port=8090
```
Note that it only remains stuck when `--drain-wait` is used. If you don't use it or also pass the `--shutdown` everything works.
Jira issue: CRDB-43732
Contributor guide
Research direction
Start with pkg/cmd/roachtest/tests/drain.go and reproduce the issue using the documented `cockroach node drain --self --drain-wait=60s` command. Trace the first invocation's drain-wait path for a secondary tenant, comparing it with a repeat invocation and with `--shutdown`; done means the initial command completes and the expected readiness response is observed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100