cockroachdb / cockroachdb/cockroach

log: decrease ExitTimeoutOnFatalLog

Open
#96,018 3 comments 0 reactions 0 assignees View on GitHub
A-storage C-bug E-quick-win T-storage
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

A `Fatalf` can stall before exiting (e.g. while sending crash report, while flushing logs) for up to `ExitTimeoutOnFatalLog` which is currently 20 seconds: https://github.com/cockroachdb/cockroach/blob/ddfeaaf30c82b7b4588d58e1455615039165f8e4/pkg/util/log/channels.go#L71

In some of the failure cases, we want the node to exit as soon as possible so that things can failover to other replicas and unblock. A recent example was handling disk stalls - we crash if an IO takes more than 30 seconds, but then if the fatal log write stalls too, we have to wait 20 more seconds.

We should reduce this to a much more aggressive timeout (on the order of a few seconds).

Jira issue: CRDB-23896

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.