cockroachdb / cockroachdb/cockroach
mmaprototype: add recovery to prevent crashes
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
`mmaprototype` currently contains `panics` and `log.Fatalf` calls that assert MMA invariants we believe should never occur. If triggered, they likely indicate a bug worth investigating, but we don’t want them to crash a production server.
- [ ] We should add panic recovery at public entry points that may encounter these failures, using https://github.com/cockroachdb/cockroach/blob/65d6d9c65852e4858115bbf1f678c0adf89b6985/pkg/util/log/logcrash/crash_reporting.go#L151 or `recover()` directly.
- [ ] Since `log.Fatalf` cannot be easily recovered from, those cases should be converted to panics so they can be caught and handled safely.
- [ ] Add metrics to catch when these panics happen.
Epic: CRDB-56265
Jira issue: CRDB-58020
Contributor guide
Research direction
Start by locating the mmaprototype public entry points and existing panic and log.Fatalf calls, then compare the recovery options in the linked crash-reporting example. Done means unsafe fatal paths can be recovered at the relevant entry points and metrics record these panics without crashing production servers.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100