cockroachdb / cockroachdb/cockroach
obs: explore options for reducing PII in core dumps
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
**Is your feature request related to a problem? Please describe.**
Core dumps are the most powerful tool we have for diagnosing frozen/hung CRDB processes, but they contain the entire process memory, including customer row data, credentials, and application-level PII. This makes them impractical to collect routinely in support workflows.
We hit this on a recent escalation where multiple nodes froze across several clusters. A core dump was the only artifact that gave us enough to find root cause (a goroutine with ~262k stack frames causing indefinite stop-the-world GC). Without it, we'd still be guessing. But the dump was ~247 GB and required explicit customer approval due to the PII exposure.
We need core dumps to be a safe and standard tool in the support toolbox.
**Describe the solution you'd like**
A way to produce core dumps that retain debugging value (goroutine stacks, runtime metadata, scheduler state) while excluding the bulk of user data.
The most promising approach: use Linux's `/proc//coredump_filter` to exclude page types that hold heap data. Most of what we need for debugging lives in stack/runtime pages, not the heap. Excluding the heap would also shrink dump sizes dramatically.
Concretely:
- Determine which `coredump_filter` bits we can clear while keeping dumps usable by Delve.
- Validate that goroutine stacks, runtime state, and scheduler metadata survive the filtering.
- Ship something like a `cockroach debug coredump` (or similar) subcommand that wraps `gcore` with the right filter, handles compression, and provides upload instructions.
**Describe alternatives you've considered**
- Userspace scrubbing. CRDB zeros block cache / SQL memory pool contents before the dump is taken. More surgical but harder to get right and maintain.
- Selective dump tooling. Build or adopt a tool that extracts only goroutine stacks and runtime metadata, skipping the heap entirely. ELF section filtering via `gcore` might work.
- Encrypted dump pipeline. Encrypt dumps at rest and in transit, limit access to engineers on the escalation. Doesn't solve the privacy problem but reduces blast radius. Orthogonal to the above.
Jira issue: CRDB-61011
Contributor guide
Assessment
This issue has not been assessed yet.