cockroachdb / cockroachdb/cockroach

roachtest,multitenant: `cockroach node drain` do not work when run for first time

Open
#133,692 1 comment 0 reactions 0 assignees View on GitHub
A-multitenancy branch-master C-bug T-db-server
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Describe the problem**

While enabling `drain/early-exit-conn-wait` roachtest in [`drain.go`](https://github.com/cockroachdb/cockroach/blob/049c30ac6956fa8f307ba2adae8b4388f5907859/pkg/cmd/roachtest/tests/drain.go) for secondary tenants (starting with shared process) I observed that `health?ready=1` do not return 50X response code. On reproducing locally I confirmed that if we trigger `cockroach node drain` _without `shutdown`_ it will remain stuck after logging,
```
I241029 06:09:30.528178 2042 server/server_controller_channel_orchestrator.go:533 ⋮ [n1,tenant-orchestration,tenant=‹test›] 165 propagate-graceful-drain task terminating
I241029 06:09:30.532992 878 server/server_controller_channel_orchestrator.go:286 ⋮ [T1,Vsystem,n1] 166 propagate-close task terminating
```

The node status is still live. But if I run the drain command again, it succeeds.

**To Reproduce**

I'm consistently able to reproduce this by running the below command,
```sh
cockroach node drain --self --drain-wait=60s --insecure --host=localhost --port=8090
```

Note that it only remains stuck when `--drain-wait` is used. If you don't use it or also pass the `--shutdown` everything works.

Jira issue: CRDB-43732

Contributor guide

Open the contributing guide

Research direction

Start with pkg/cmd/roachtest/tests/drain.go and reproduce the issue using the documented `cockroach node drain --self --drain-wait=60s` command. Trace the first invocation's drain-wait path for a secondary tenant, comparing it with a repeat invocation and with `--shutdown`; done means the initial command completes and the expected readiness response is observed.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.