dev-server crash on AddOrUpdateRemoteCluster RPC with invalid frontend address

Expected Behavior

Server rejects the RPC with an error, but continues to run.

Actual Behavior

Server crashes (that is, temporal server start-dev exits with error code 1).

Error log message: {"level":"fatal",...,"msg":"Invalid rpcAddress for remote cluster","error":"address asdf: missing port in address","logging-call-at":"rpc.go:191", ...

Stack trace:

go.temporal.io/server/common/log.(*zapLogger).Fatal
        go.temporal.io/server@v1.22.4/common/log/zap_logger.go:180
go.temporal.io/server/common/rpc.(*RPCFactory).CreateRemoteFrontendGRPCConnection
        go.temporal.io/server@v1.22.4/common/rpc/rpc.go:191
go.temporal.io/server/client.(*rpcClientFactory).NewRemoteAdminClientWithTimeout
        go.temporal.io/server@v1.22.4/client/clientfactory.go:195
go.temporal.io/server/service/frontend.(*OperatorHandlerImpl).AddOrUpdateRemoteCluster
        go.temporal.io/server@v1.22.4/service/frontend/operator_handler.go:607
go.temporal.io/api/operatorservice/v1._OperatorService_AddOrUpdateRemoteCluster_Handler.func1
        go.temporal.io/api@v1.26.0/operatorservice/v1/service.pb.go:347
go.temporal.io/server/common/rpc/interceptor.(*RetryableInterceptor).Intercept.func1
        go.temporal.io/server@v1.22.4/common/rpc/interceptor/retry.go:63
go.temporal.io/server/common/backoff.ThrottleRetryContext
        go.temporal.io/server@v1.22.4/common/backoff/retry.go:145
go.temporal.io/server/common/rpc/interceptor.(*RetryableInterceptor).Intercept
        go.temporal.io/server@v1.22.4/common/rpc/interceptor/retry.go:67
google.golang.org/grpc.getChainUnaryHandler.func1
        google.golang.org/grpc@v1.59.0/server.go:1163
go.temporal.io/server/common/rpc/interceptor.(*CallerInfoInterceptor).Intercept
        go.temporal.io/server@v1.22.4/common/rpc/interceptor/caller_info.go:80
google.golang.org/grpc.getChainUnaryHandler.func1
        google.golang.org/grpc@v1.59.0/server.go:1163
go.temporal.io/server/common/rpc/interceptor.(*SDKVersionInterceptor).Intercept
        go.temporal.io/server@v1.22.4/common/rpc/interceptor/sdk_version.go:69
google.golang.org/grpc.getChainUnaryHandler.func1
        google.golang.org/grpc@v1.59.0/server.go:1163
go.temporal.io/server/common/rpc/interceptor.(*RateLimitInterceptor).Intercept
        go.temporal.io/server@v1.22.4/common/rpc/interceptor/rate_limit.go:88
google.golang.org/grpc.getChainUnaryHandler.func1
        google.golang.org/grpc@v1.59.0/server.go:1163
go.temporal.io/server/common/rpc/interceptor.(*NamespaceRateLimitInterceptor).Intercept
        go.temporal.io/server@v1.22.4/common/rpc/interceptor/namespace_rate_limit.go:93
google.golang.org/grpc.getChainUnaryHandler.func1
        google.golang.org/grpc@v1.59.0/server.go:1163
go.temporal.io/server/common/rpc/interceptor.(*ConcurrentRequestLimitInterceptor).Intercept
        go.temporal.io/server@v1.22.4/common/rpc/interceptor/concurrent_request_limit.go:121
google.golang.org/grpc.getChainUnaryHandler.func1
        google.golang.org/grpc@v1.59.0/server.go:1163
go.temporal.io/server/common/rpc/interceptor.(*NamespaceValidatorInterceptor).StateValidationIntercept
        go.temporal.io/server@v1.22.4/common/rpc/interceptor/namespace_validator.go:194
google.golang.org/grpc.getChainUnaryHandler.func1
        google.golang.org/grpc@v1.59.0/server.go:1163
go.temporal.io/server/common/rpc/interceptor.(*TelemetryInterceptor).UnaryIntercept
        go.temporal.io/server@v1.22.4/common/rpc/interceptor/telemetry.go:165
google.golang.org/grpc.getChainUnaryHandler.func1
        google.golang.org/grpc@v1.59.0/server.go:1163
go.temporal.io/server/service/frontend.(*RedirectionInterceptor).Intercept
        go.temporal.io/server@v1.22.4/service/frontend/redirection_interceptor.go:180
google.golang.org/grpc.getChainUnaryHandler.func1
        google.golang.org/grpc@v1.59.0/server.go:1163
go.temporal.io/server/common/authorization.(*interceptor).Interceptor
        go.temporal.io/server@v1.22.4/common/authorization/interceptor.go:158
google.golang.org/grpc.getChainUnaryHandler.func1
        google.golang.org/grpc@v1.59.0/server.go:1163
go.temporal.io/server/service/frontend.GrpcServerOptionsProvider.NewServerMetricsContextInjectorInterceptor.func1
        go.temporal.io/server@v1.22.4/common/metrics/grpc.go:66
google.golang.org/grpc.getChainUnaryHandler.func1
        google.golang.org/grpc@v1.59.0/server.go:1163
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.UnaryServerInterceptor.func1
        go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc@v0.42.0/interceptor.go:344
google.golang.org/grpc.getChainUnaryHandler.func1
        google.golang.org/grpc@v1.59.0/server.go:1163
go.temporal.io/server/common/rpc/interceptor.(*NamespaceLogInterceptor).Intercept
        go.temporal.io/server@v1.22.4/common/rpc/interceptor/namespace_logger.go:84
google.golang.org/grpc.getChainUnaryHandler.func1
        google.golang.org/grpc@v1.59.0/server.go:1163
go.temporal.io/server/common/rpc/interceptor.(*NamespaceValidatorInterceptor).NamespaceValidateIntercept
        go.temporal.io/server@v1.22.4/common/rpc/interceptor/namespace_validator.go:111
google.golang.org/grpc.getChainUnaryHandler.func1
        google.golang.org/grpc@v1.59.0/server.go:1163
go.temporal.io/server/common/rpc.ServiceErrorInterceptor
        go.temporal.io/server@v1.22.4/common/rpc/grpc.go:145
google.golang.org/grpc.NewServer.chainUnaryServerInterceptors.chainUnaryInterceptors.func1
        google.golang.org/grpc@v1.59.0/server.go:1154
go.temporal.io/api/operatorservice/v1._OperatorService_AddOrUpdateRemoteCluster_Handler
        go.temporal.io/api@v1.26.0/operatorservice/v1/service.pb.go:349
google.golang.org/grpc.(*Server).processUnaryRPC
        google.golang.org/grpc@v1.59.0/server.go:1343
google.golang.org/grpc.(*Server).handleStream
        google.golang.org/grpc@v1.59.0/server.go:1737
google.golang.org/grpc.(*Server).serveStreams.func1.1
        google.golang.org/grpc@v1.59.0/server.go:986

Steps to Reproduce the Problem

1. temporal server start-dev
2. temporal operator cluster upsert --frontend-address=asdf
3. Server process exits with error code 1.

(I'm not sure if this is repro'able on an actual server, but the crash is coming from within server code, so I would assume so—hence why I'm filing the issue here instead of against the CLI.)

Specifications

• Version: temporal version 0.11.0 (server 1.22.4) (ui 2.21.3)
• Platform: Mac ARM

Reproduce the failure with temporal server start-dev and temporal operator cluster upsert --frontend-address=asdf. Read common/rpc/rpc.go at line 191 and the call from service/frontend/operator_handler.go at line 607. Done means the invalid RPC returns an error while the server continues running, with coverage for this behavior.