loggging/exception stack trace not capturing underlying errors

I configured temporal 1.21 using k8s and s3 archival on EKS. apparently my s3 access key was in correct. when i started temporal it started fine, but when i

Expected Behavior

Error should have underlying cause

Actual Behavior

Archival related errors do not get shown up in create name space calls

Steps to Reproduce the Problem

1. configure temporal on eks cluster with s3 archival
2. do not configure s3 access key/or invalid key
3. start temporal, it gets started successfully
4. create a namespace from tctl or client library and you will get 403 without any hint that it origniated from archival/s3.

Specifications

• Version: 1.2.1
• Platform: k8s/eks

here is the stack trace :

bash-5.0# tctl --namespace samples-namespace namespace register
Error: Register namespace operation failed.
Error Details: rpc error: code = Unknown desc = Forbidden: Forbidden
status code: 403, request id: 250DCE81B9026FA0, host id: bTkqpS/CChOZrwZw14CL1p8mOfKlnEMU66Pm25Mx33DxGk6a+KD6xLvdso3cX7xhjAopvQ/dUGA=
(‘export TEMPORAL_CLI_SHOW_STACKS=1’ to see stack traces)

in the front end i see the following in logs

{“level”:“error”,“ts”:“2020-11-11T11:22:30.917Z”,“msg”:“Unknown error”,“service”:“frontend”,“error”:“Forbidden: Forbidden\n\tstatus code: 403, request id: C486AFCD8246BCA6, host id: p3YC1NdyUDJbVfGd6OrQ8IYuLcOmqSYpQaauroLUyQUoF7qkekVFxR+mQrX5bgQC3DVf2/L7BWI=”,“logging-call-at”:“workflowHandler.go:3399”,“stacktrace”:“go.temporal.io/server/common/log/loggerimpl.(*loggerImpl).Error\n\t/temporal/common/log/loggerimpl/logger.go:138\ngo.temporal.io/server/service/frontend.(*WorkflowHandler).error\n\t/temporal/service/frontend/workflowHandler.go:3399\ngo.temporal.io/server/service/frontend.(*WorkflowHandler).RegisterNamespace\n\t/temporal/service/frontend/workflowHandler.go:252\ngo.temporal.io/server/service/frontend.(*DCRedirectionHandlerImpl).RegisterNamespace\n\t/temporal/service/frontend/dcRedirectionHandler.go:177\ngo.temporal.io/server/service/frontend.(*AccessControlledWorkflowHandler).RegisterNamespace\n\t/temporal/service/frontend/accessControlledHandler.go:469\ngo.temporal.io/server/service/frontend.(*WorkflowNilCheckHandler).RegisterNamespace\n\t/temporal/service/frontend/workflowNilCheckHandler.go:62\ngo.temporal.io/api/workflowservice/v1._WorkflowService_RegisterNamespace_Handler.func1\n\t/go/pkg/mod/go.temporal.io/api@v1.1.0/workflowservice/v1/service.pb.go:957\ngo.temporal.io/server/common/rpc.Interceptor\n\t/temporal/common/rpc/interceptor.go:35\ngo.temporal.io/api/workflowservice/v1._WorkflowService_RegisterNamespace_Handler\n\t/go/pkg/mod/go.temporal.io/api@v1.1.0/workflowservice/v1/service.pb.go:959\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.32.0/server.go:1194\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.32.0/server.go:1517\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.2\n\t/go/pkg/mod/google.golang.org/grpc@v1.32.0/server.go:859”}

Start at service/frontend/workflowHandler.go around RegisterNamespace and trace the archival error path described by the stack trace. Reproduce with invalid EKS archival credentials using tctl or a client, then verify that the namespace-registration error exposes the underlying archival failure instead of only a bare 403.