cockroachdb / cockroachdb/cockroach
server,util/log: file sinks without exit-on-error set may never recover from errors
Open
A-logging
branch-master
branch-release-24.3
C-bug
O-24.3-scale-testing
O-25.4-scale-testing
O-testcluster
P-2
T-supportability
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
**Describe the problem**
This is a placeholder that @stevendanna will add more information to.
- The file sink uses a bufio.Writer internally.
- When a bufio.Writer encounters an error, from the underlying io.Writer, it returns that error on all future calls.
- Unless a fatal message arrives or the sink is marked as "critical" (exit-on-error set in the configuration), this error is only ever logged.
As a result, we have observed a CRDB node continue to fail to flush log entries several days after the device holding the log directory ran out of space and was then cleaned up.
Jira issue: CRDB-44264
Contributor guide
Assessment
This issue has not been assessed yet.