cockroachdb / cockroachdb/cockroach
Disk stall FATAL log entry is tens of MB due to full goroutine dump
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
## Describe the problem
When a node hits a disk stall, the process logs a single FATAL log entry that can be extremely large (tens of MB per occurrence). Almost all of that size comes from the full goroutine dump (thousands of goroutines' stack traces) attached to that one entry.
This causes several issues:
1. **Usability** – The entry is too large to tail, grep, or ship to log aggregators in a practical way, so the actual disk-stall message and I/O stats are hard to use in normal workflows.
2. **No separation of severity** – The critical part (disk stall message + I/O stats) is bundled with very verbose debug-style data (full process stacks) in a single record.
We'd like to see either:
- **Option A:** Emit the disk-stall FATAL as a normal-sized entry (message + I/O stats only), and emit the full goroutine dump as a **separate** log entry (e.g. DEV/DEBUG or a dedicated "stacks" entry), so the FATAL itself stays small and stacks can be filtered or omitted when sharing logs, or
- **Option B:** By default, **do not** attach the full goroutine dump to disk-stall FATALs; keep the FATAL to the disk-stall message and I/O stats. Full stacks could be opt-in (e.g. via an env var or logging config) for rare deep debugging.
---
## To Reproduce
**What did you do?**
1. Run a CockroachDB cluster (any topology).
2. Induce or wait for a disk stall (e.g. storage unable to sync within the stall threshold, such as 40s).
3. The node logs a FATAL and exits.
4. Inspect the log file for that node.
**If possible, provide steps to reproduce the behavior:**
1. Set up a CockroachDB cluster with storage that can be slowed or stalled (e.g. underlying disk or network storage that can be throttled or delayed).
2. Trigger a disk stall (e.g. sync on a probe file exceeding the stall threshold).
3. Open the node's log directory and open the most recent log file.
4. Observe a single FATAL log entry that is tens of MB in size, almost entirely from the `stacks` field (full goroutine dump).
---
## Expected behavior
- **Option A:** The disk-stall event is represented by:
- One FATAL log entry containing only the disk-stall message and I/O stats (small, easy to grep/ship).
- A separate log entry (e.g. DEBUG or a dedicated type) containing the full goroutine dump, so the FATAL itself has no PII-heavy stacks and can be shared safely by default.
- **Option B:** By default, the disk-stall FATAL contains only the disk-stall message and I/O stats (no full goroutine dump). Full stacks are available only when explicitly enabled (e.g. via configuration or env var).
In both cases, the default or "normal" disk-stall FATAL should be a single, small log line (hundreds of bytes to low KB), not tens of MB.
---
## Additional data / screenshots
In the customer reported example, the breakdown of a single log line is as follows:
## DEV:FATAL log entry – section sizes
**Total record:** 33,295,789 bytes, 33,294,589 characters
### Top-level JSON keys
- **metadata** (channel, file, goroutine, line, severity, tags, timestamp, version, etc.): ~150 bytes, 0.0%
- **message:** 102,701 bytes, 0.3%
- **stacks:** 33,192,800 bytes, 99.7%
### message subsections
- First line (disk stall summary): 95 bytes, 0.0%
- I/O stats table (repeated rows): 101,958 bytes, 0.3%
- Other (headers, etc.): 343 bytes, 0.0%
### stacks (per goroutine)
- 5,774 goroutines in this single log entry.
- First ~20: a few hundred bytes each.
- Remaining 5,754: ~32.6 MB total (almost the entire entry).
---
## Environment
- CockroachDB version: v24.3.17
- Server OS: Linux.
- Log format: JSON (one JSON object per line, records split by `}\n{`).
---
## Additional context
**Impact**
- Operators cannot practically tail or grep for "disk stall" without handling multi-MB lines.
- Log aggregation/ingestion can be expensive or broken for these entries.
- Logs are very large, contributing to diskIO pressure and making it very hard to parse / use for support
We're filing this to request a change so that disk-stall FATALs are small by default and full stacks are either separate (Option A) or opt-in (Option B), with zero PII in the default FATAL payload.
Jira issue: CRDB-59190
Contributor guide
Assessment
This issue has not been assessed yet.