[Bug] Python client not able to connect to self-hosted Temporal server via proxy using authorization header

What are you really trying to do?

We’re trying to connect Python client to our self-hosted Temporal server via a proxy using an authorization header.

Describe the bug

We see that it is able to connect to the worker at startup, but after exactly 60 seconds, it fails with the below error. Note that we do see some transient 503 errors while connecting to the server; but after that it’s able to connect successfully (verified in server logs).

Minimal Reproduction

Here’ the set of errors we see:
1.

Error: Client connection not established Failed client connect: `get_system_info` call error after connection: Status { code: PermissionDenied, message: "encountered c2c service api error response 503", metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Wed, 19 Mar 2025 02:11:39 GMT"} }, source: None }.
Traceback (most recent call last):
  File "/home/temporal/temporal_sample_worker_python/src/run_worker.py", line 202, in <module>
    asyncio.run(main())
  File "/usr/lib64/python3.9/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/lib64/python3.9/asyncio/base_events.py", line 647, in run_until_complete
    return future.result()
  File "/home/temporal/temporal_sample_worker_python/src/run_worker.py", line 157, in main
    await conn.create_client()
  File "/home/temporal/temporal_sample_worker_python/src/utils.py", line 105, in create_client
    self.client = await Client.connect("dev.api.data.falcon.sfdcfc.net:7443",
  File "/usr/local/lib64/python3.9/site-packages/temporalio/client.py", line 176, in connect
    await temporalio.service.ServiceClient.connect(connect_config),
  File "/usr/local/lib64/python3.9/site-packages/temporalio/service.py", line 209, in connect
    return await _BridgeServiceClient.connect(config)
  File "/usr/local/lib64/python3.9/site-packages/temporalio/service.py", line 1117, in connect
    await client._connected_client()
  File "/usr/local/lib64/python3.9/site-packages/temporalio/service.py", line 1130, in _connected_client
    self._bridge_client = await temporalio.bridge.client.Client.connect(
  File "/usr/local/lib64/python3.9/site-packages/temporalio/bridge/client.py", line 97, in connect
    await temporalio.bridge.temporal_sdk_bridge.connect_client(
RuntimeError: Failed client connect: `get_system_info` call error after connection: Status { code: PermissionDenied, message: "encountered c2c service api error response 503", metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Wed, 19 Mar 2025 02:11:39 GMT"} }, source: None }

2025-03-19T02:17:48.974543Z  WARN temporal_sdk_core::worker::workflow::wft_poller: Error while polling for workflow tasks error=Status { code: Unimplemented, metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Wed, 19 Mar 2025 02:17:48 GMT"} }, source: None }
2025-03-19T02:17:48.974547Z ERROR temporal_sdk_core::worker::workflow::workflow_stream: Workflow processing encountered fatal error and must shut down TonicError(Status { code: Unimplemented, metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Wed, 19 Mar 2025 02:17:48 GMT"} }, source: None })
2025-03-19T02:17:48.974551Z ERROR temporal_sdk_core::worker::workflow::workflow_stream: Workflow processing encountered fatal error and must shut down TonicError(Status { code: Unimplemented, metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Wed, 19 Mar 2025 02:17:48 GMT"} }, source: None })
2025-03-19 02:17:48,975 - ERROR - Worker failed, shutting down
Traceback (most recent call last):
  File "/usr/local/lib64/python3.9/site-packages/temporalio/worker/_workflow.py", line 154, in run
    act = await self._bridge_worker().poll_workflow_activation()
  File "/usr/local/lib64/python3.9/site-packages/temporalio/bridge/worker.py", line 147, in poll_workflow_activation
    await self._ref.poll_workflow_activation()
RuntimeError: Poll failure: Unhandled grpc error when polling: Status { code: Unimplemented, metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Wed, 19 Mar 2025 02:17:48 GMT"} }, source: None }

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib64/python3.9/site-packages/temporalio/worker/_workflow.py", line 164, in run
    raise RuntimeError("Workflow worker failed") from err
RuntimeError: Workflow worker failed
2025-03-19 02:17:48,975 - INFO - Beginning worker shutdown, will wait 0:00:00 before cancelling activities

Traceback (most recent call last):
  File "/usr/local/lib64/python3.9/site-packages/temporalio/worker/_workflow.py", line 154, in run
    act = await self._bridge_worker().poll_workflow_activation()
  File "/usr/local/lib64/python3.9/site-packages/temporalio/bridge/worker.py", line 147, in poll_workflow_activation
    await self._ref.poll_workflow_activation()
RuntimeError: Poll failure: Unhandled grpc error when polling: Status { code: Unimplemented, metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Wed, 19 Mar 2025 02:17:48 GMT"} }, source: None }

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/temporal/temporal_sample_worker_python/src/run_worker.py", line 202, in <module>
    asyncio.run(main())
  File "/usr/lib64/python3.9/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/lib64/python3.9/asyncio/base_events.py", line 647, in run_until_complete
    return future.result()
  File "/home/temporal/temporal_sample_worker_python/src/run_worker.py", line 179, in main
    await run_worker(client, taskqueue_name, args.worker_type)
  File "/home/temporal/temporal_sample_worker_python/src/run_worker.py", line 142, in run_worker
    await worker.run()
  File "/usr/local/lib64/python3.9/site-packages/temporalio/worker/_worker.py", line 571, in run
    raise exception
  File "/usr/local/lib64/python3.9/site-packages/temporalio/worker/_workflow.py", line 164, in run
    raise RuntimeError("Workflow worker failed") from err
RuntimeError: Workflow worker failed

Environment/Versions

• Temporal Version: Server v1.26.2, SDK v1.10.0

Additional context

Our issue looks similar to this this open bug: [Bug] Worker hangs after polling workflow task queue · Issue #631 · temporalio/sdk-python · GitHub
Wanted to check if it is same issue and if not we need help in understanding the fixing it.

Start with the Client.connect call in the reported src/utils.py and the worker flow in src/run_worker.py, then reproduce the failure through the proxy using the stated server and SDK versions. Compare the 60-second connection failure and subsequent polling errors with issue #631; done means the Python worker remains connected and polls successfully with the authorization header.