cockroachdb / cockroachdb/cockroach
log: include stack trace when exiting node due to log error
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
**Describe the problem**
In a recent incident, nodes were exiting while outputting logs due to this function being called:
https://github.com/cockroachdb/cockroach/blob/049c30ac6956fa8f307ba2adae8b4388f5907859/pkg/util/log/exit_override.go#L81
Since SRE doesn't work a lot with the DB code, it wasn't obvious to them why the nodes were exiting and took some time before realizing that the nodes were exiting due to errors being thrown from the logging sink.
SRE mentioned that one thing they look out for in the case of node crashes is stack traces and requested that we include a stack trace when exiting here to make more apparent that nodes are exiting due to log sink issues.
Jira issue: CRDB-53952
Contributor guide
Assessment
This issue has not been assessed yet.