[PipeCD-Server] Server should not crash when panic occurs
@theycallmeaabie is already working on this.
Since Apr 8, 2026.
- Dominant language
- Go
- Stars
- 1.4k
- Forks
- 364
- Avg merge
- 1d 4h
- Merged PRs (30d)
- 84
Description
What would you like to be added: An interceptor to help server recovery from panic
Why is this needed: pipecd server crashes and restarts when panic occurs, we should avoid this happens in production environment
There are 2 ways to do this:
- directly use https://github.com/grpc-ecosystem/go-grpc-middleware/blob/v1.4.0/recovery/interceptors.go
- implement by our own to avoid dependency
// PanicRecoveryUnaryServerInterceptor returns a gRPC unary interceptor that
// recovers from panics in request handlers to prevent the server from crashing.
func PanicRecoveryUnaryServerInterceptor(logger *zap.Logger) grpc.UnaryServerInterceptor {
return func(ctx context.Context, req interface{}, info *grpc.UnaryServerInfo, handler grpc.UnaryHandler) (_ interface{}, err error) {
panicked := true
defer func() {
if r := recover(); r != nil || panicked {
var panicErr error
switch v := r.(type) {
case error:
panicErr = v
case nil:
panicErr = fmt.Errorf("runtime.Goexit() called")
default:
panicErr = fmt.Errorf("%v", v)
}
logger.Error("panic recovered in gRPC handler",
zap.String("method", info.FullMethod),
zap.Error(panicErr),
zap.Stack("stack"),
)
err = status.Error(codes.Internal, "internal server error")
}
}()
resp, err := handler(ctx, req)
panicked = false
return resp, err
}
}
Explain a little bit for the usage of panicked: when panic occurs, recover() returns the value of panic (panic != nil) but when Goexit, recover() return nil -> the condition r != nil won't detect it
panicked flag helps us distinguishing "handler returns normally" or "handler is terminated by Goexit" (this technique I learn from https://github.com/grpc-ecosystem/go-grpc-middleware/blob/v1.4.0/recovery/interceptors.go)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.