cockroachdb / cockroachdb/cockroach
sql/stmtdiagnostics: statement and transaction diagnostics pollers log repeatedly during drain
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
**Describe the problem**
The statement and transaction diagnostics pollers in [`StartPolling`](https://github.com/cockroachdb/cockroach/blob/master/pkg/sql/stmtdiagnostics/statement_diagnostics.go#L832) repeatedly log warnings during the drain-to-quiesce window:
```
error polling for transaction diagnostics requests: txn-diag-poll: cannot acquire lease when draining
error polling for statement diagnostics requests: stmt-diag-poll: cannot acquire lease when draining
```
The polling context is cancelled via `stopper.WithCancelOnQuiesce`, but the lease manager starts rejecting acquisitions with `errLeaseManagerIsDraining` earlier — when `SetDraining(true)` is called during the drain phase, before quiesce. The polling loop keeps running, the queries keep failing, and the warnings keep logging until the stopper finally quiesces.
If something else is blocking quiesce from completing, these warnings log indefinitely, as seen in the wild on a 26.1 cluster during pod deletion.
**To Reproduce**
1. Start a CockroachDB cluster
2. Issue SIGTERM / initiate drain on a node
3. Observe repeated `cannot acquire lease when draining` warnings from `stmt-diag-poll` and `txn-diag-poll` in the logs during the drain-to-quiesce window
**Expected behavior**
The pollers should stop polling once drain starts, rather than continuing to attempt queries that are guaranteed to fail until quiesce.
**Additional context**
Related PRs that fixed similar drain-time polling issues:
- #160658 — fixed infinite retries in `EnsureBatch` during quiesce
- #160656 — fixed `CaptureIndexUsageStats` blocking shutdown
A possible fix would be to detect `errLeaseManagerIsDraining` in the `poll()` closure and treat it as terminal, stopping the polling loop rather than just logging and retrying.
Jira issue: CRDB-62375
Contributor guide
Assessment
This issue has not been assessed yet.