cockroachdb / cockroachdb/cockroach

log: include stack trace when exiting node due to log error

Open
#152,636 2 comments 0 reactions 0 assignees View on GitHub
C-bug O-postmortem T-supportability
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

**Describe the problem**

In a recent incident, nodes were exiting while outputting logs due to this function being called:
https://github.com/cockroachdb/cockroach/blob/049c30ac6956fa8f307ba2adae8b4388f5907859/pkg/util/log/exit_override.go#L81

Since SRE doesn't work a lot with the DB code, it wasn't obvious to them why the nodes were exiting and took some time before realizing that the nodes were exiting due to errors being thrown from the logging sink.

SRE mentioned that one thing they look out for in the case of node crashes is stack traces and requested that we include a stack trace when exiting here to make more apparent that nodes are exiting due to log sink issues.

Jira issue: CRDB-53952

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.