[Bug] Panic when failing to start worker due to being unauthorized

What are you really trying to do?

Trying to bring up a worker process for the first time in our production environment.

Describe the bug

There's a panic here. There probably shouldn't be? (It's probably not super important, but would be nice to handle in a better way.)

2024-02-23T16:29:13.189124Z ERROR temporal_sdk_core::worker::workflow::workflow_stream: Workflow processing encountered fatal error and must shut down TonicError(Status { code: PermissionDenied, message: "Request unauthorized.", details: b"\x08\x07\x12\x15Request unauthorized.\x1a\x8c\x01\nHtype.googleapis.com/temporal.api.errordetails.v1.PermissionDeniedFailure\x12@\n>caller does not have required permissions for namespace 'prod'", metadata: MetadataMap { headers: {"content-type": "application/grpc", "server": "temporal", "date": "Fri, 23 Feb 2024 16:29:12 GMT"} }, source: None })
2024-02-23T16:29:13.190812Z ERROR panic: shared::tracers: panic: panicked at /Users/djc/.cargo/git/checkouts/sdk-core-3d5ba50f52602518/42e2a94/core/src/worker/workflow/mod.rs:231:34:
Activation processor channel not dropped: SendError { .. } location="/Users/djc/.cargo/git/checkouts/sdk-core-3d5ba50f52602518/42e2a94/core/src/worker/workflow/mod.rs:231:34" payload="unknown"
thread 'workflow-processing' panicked at /Users/djc/.cargo/git/checkouts/sdk-core-3d5ba50f52602518/42e2a94/core/src/worker/workflow/mod.rs:231:34:
Activation processor channel not dropped: SendError { .. }
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Error: Unhandled grpc error when activity polling: Status { code: PermissionDenied, message: "Request unauthorized.", details: b"\x08\x07\x12\x15Request unauthorized.\x1a\x8c\x01\nHtype.googleapis.com/temporal.api.errordetails.v1.PermissionDeniedFailure\x12@\n>caller does not have required permissions for namespace 'prod'", metadata: MetadataMap { headers: {"content-type": "application/grpc", "server": "temporal", "date": "Fri, 23 Feb 2024 16:29:12 GMT"} }, source: None }

Minimal Reproduction

Minimal Rust SDK worker setup along the lines of the example in the crate root, using mTLS to connect to Temporal Cloud.

Environment/Versions

• OS and processor: M1 Mac
• Temporal Version: sdk-core @ 42e2a94baad618ca352b9faad5879dfb59ff4a04
• Are you using Docker or Kubernetes or building Temporal from source? See above.

Start at core/src/worker/workflow/mod.rs:231 and reproduce the unauthorized worker startup using the minimal Rust SDK setup with mTLS. Trace how the PermissionDenied error reaches workflow processing and activity polling; done means the unauthorized startup reports the error without the Activation processor channel panic.