cockroachdb / cockroachdb/cockroach
log: decrease ExitTimeoutOnFatalLog
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
A `Fatalf` can stall before exiting (e.g. while sending crash report, while flushing logs) for up to `ExitTimeoutOnFatalLog` which is currently 20 seconds: https://github.com/cockroachdb/cockroach/blob/ddfeaaf30c82b7b4588d58e1455615039165f8e4/pkg/util/log/channels.go#L71
In some of the failure cases, we want the node to exit as soon as possible so that things can failover to other replicas and unblock. A recent example was handling disk stalls - we crash if an IO takes more than 30 seconds, but then if the fatal log write stalls too, we have to wait 20 more seconds.
We should reduce this to a much more aggressive timeout (on the order of a few seconds).
Jira issue: CRDB-23896
Contributor guide
Assessment
This issue has not been assessed yet.