cockroachdb / cockroachdb/cockroach

mmaprototype: add recovery to prevent crashes

Open
#159,786 0 comments 0 reactions 0 assignees View on GitHub
A-kv-distribution C-enhancement T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

`mmaprototype` currently contains `panics` and `log.Fatalf` calls that assert MMA invariants we believe should never occur. If triggered, they likely indicate a bug worth investigating, but we don’t want them to crash a production server.

- [ ] We should add panic recovery at public entry points that may encounter these failures, using https://github.com/cockroachdb/cockroach/blob/65d6d9c65852e4858115bbf1f678c0adf89b6985/pkg/util/log/logcrash/crash_reporting.go#L151 or `recover()` directly.
- [ ] Since `log.Fatalf` cannot be easily recovered from, those cases should be converted to panics so they can be caught and handled safely.
- [ ] Add metrics to catch when these panics happen.

Epic: CRDB-56265

Jira issue: CRDB-58020

Contributor guide

Open the contributing guide

Research direction

Start by locating the mmaprototype public entry points and existing panic and log.Fatalf calls, then compare the recovery options in the linked crash-reporting example. Done means unsafe fatal paths can be recovered at the relevant entry points and metrics record these panics without crashing production servers.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.