pipe-cd / pipe-cd/pipecd

[PipeCD-Server] Server should not crash when panic occurs

Open
#6,612 2 comments 0 reactions 1 assignee View on GitHub

@theycallmeaabie is already working on this.

Since Apr 8, 2026.

kind/enhancement
Dominant language
Go
Stars
1.4k
Forks
364
Avg merge
1d 4h
Merged PRs (30d)
84

Description

What would you like to be added: An interceptor to help server recovery from panic

Why is this needed: pipecd server crashes and restarts when panic occurs, we should avoid this happens in production environment

There are 2 ways to do this:

  1. directly use https://github.com/grpc-ecosystem/go-grpc-middleware/blob/v1.4.0/recovery/interceptors.go
  2. implement by our own to avoid dependency
// PanicRecoveryUnaryServerInterceptor returns a gRPC unary interceptor that
// recovers from panics in request handlers to prevent the server from crashing.
func PanicRecoveryUnaryServerInterceptor(logger *zap.Logger) grpc.UnaryServerInterceptor {
	return func(ctx context.Context, req interface{}, info *grpc.UnaryServerInfo, handler grpc.UnaryHandler) (_ interface{}, err error) {
		panicked := true
		defer func() {
			if r := recover(); r != nil || panicked {
				var panicErr error
				switch v := r.(type) {
				case error:
					panicErr = v
				case nil:
					panicErr = fmt.Errorf("runtime.Goexit() called")
				default:
					panicErr = fmt.Errorf("%v", v)
				}

				logger.Error("panic recovered in gRPC handler",
					zap.String("method", info.FullMethod),
					zap.Error(panicErr),
					zap.Stack("stack"),
				)

				err = status.Error(codes.Internal, "internal server error")
			}
		}()
		resp, err := handler(ctx, req)
		panicked = false
		return resp, err
	}
}

Explain a little bit for the usage of panicked: when panic occurs, recover() returns the value of panic (panic != nil) but when Goexit, recover() return nil -> the condition r != nil won't detect it

panicked flag helps us distinguishing "handler returns normally" or "handler is terminated by Goexit" (this technique I learn from https://github.com/grpc-ecosystem/go-grpc-middleware/blob/v1.4.0/recovery/interceptors.go)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.