[Bug] Python client not able to connect to self-hosted Temporal server via proxy using authorization header

Comienza con la llamada a Client.connect en el src/utils.py indicado y con el flujo del worker en src/run_worker.py; después reproduce el fallo a través del proxy usando las versiones indicadas del servidor y del SDK. Compara el fallo de conexión de 60 segundos y los errores de polling posteriores con el issue #631; se considera terminado cuando el worker de Python permanece conectado y realiza el polling correctamente con el encabezado de autorización.

What are you really trying to do?

We’re trying to connect Python client to our self-hosted Temporal server via a proxy using an authorization header.

Describe the bug

We see that it is able to connect to the worker at startup, but after exactly 60 seconds, it fails with the below error. Note that we do see some transient 503 errors while connecting to the server; but after that it’s able to connect successfully (verified in server logs).

Minimal Reproduction

Here’ the set of errors we see:
1.

Error: Client connection not established Failed client connect: `get_system_info` call error after connection: Status { code: PermissionDenied, message: "encountered c2c service api error response 503", metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Wed, 19 Mar 2025 02:11:39 GMT"} }, source: None }.
Traceback (most recent call last):
  File "/home/temporal/temporal_sample_worker_python/src/run_worker.py", line 202, in <module>
    asyncio.run(main())
  File "/usr/lib64/python3.9/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/lib64/python3.9/asyncio/base_events.py", line 647, in run_until_complete
    return future.result()
  File "/home/temporal/temporal_sample_worker_python/src/run_worker.py", line 157, in main
    await conn.create_client()
  File "/home/temporal/temporal_sample_worker_python/src/utils.py", line 105, in create_client
    self.client = await Client.connect("dev.api.data.falcon.sfdcfc.net:7443",
  File "/usr/local/lib64/python3.9/site-packages/temporalio/client.py", line 176, in connect
    await temporalio.service.ServiceClient.connect(connect_config),
  File "/usr/local/lib64/python3.9/site-packages/temporalio/service.py", line 209, in connect
    return await _BridgeServiceClient.connect(config)
  File "/usr/local/lib64/python3.9/site-packages/temporalio/service.py", line 1117, in connect
    await client._connected_client()
  File "/usr/local/lib64/python3.9/site-packages/temporalio/service.py", line 1130, in _connected_client
    self._bridge_client = await temporalio.bridge.client.Client.connect(
  File "/usr/local/lib64/python3.9/site-packages/temporalio/bridge/client.py", line 97, in connect
    await temporalio.bridge.temporal_sdk_bridge.connect_client(
RuntimeError: Failed client connect: `get_system_info` call error after connection: Status { code: PermissionDenied, message: "encountered c2c service api error response 503", metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Wed, 19 Mar 2025 02:11:39 GMT"} }, source: None }

2025-03-19T02:17:48.974543Z  WARN temporal_sdk_core::worker::workflow::wft_poller: Error while polling for workflow tasks error=Status { code: Unimplemented, metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Wed, 19 Mar 2025 02:17:48 GMT"} }, source: None }
2025-03-19T02:17:48.974547Z ERROR temporal_sdk_core::worker::workflow::workflow_stream: Workflow processing encountered fatal error and must shut down TonicError(Status { code: Unimplemented, metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Wed, 19 Mar 2025 02:17:48 GMT"} }, source: None })
2025-03-19T02:17:48.974551Z ERROR temporal_sdk_core::worker::workflow::workflow_stream: Workflow processing encountered fatal error and must shut down TonicError(Status { code: Unimplemented, metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Wed, 19 Mar 2025 02:17:48 GMT"} }, source: None })
2025-03-19 02:17:48,975 - ERROR - Worker failed, shutting down
Traceback (most recent call last):
  File "/usr/local/lib64/python3.9/site-packages/temporalio/worker/_workflow.py", line 154, in run
    act = await self._bridge_worker().poll_workflow_activation()
  File "/usr/local/lib64/python3.9/site-packages/temporalio/bridge/worker.py", line 147, in poll_workflow_activation
    await self._ref.poll_workflow_activation()
RuntimeError: Poll failure: Unhandled grpc error when polling: Status { code: Unimplemented, metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Wed, 19 Mar 2025 02:17:48 GMT"} }, source: None }

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib64/python3.9/site-packages/temporalio/worker/_workflow.py", line 164, in run
    raise RuntimeError("Workflow worker failed") from err
RuntimeError: Workflow worker failed
2025-03-19 02:17:48,975 - INFO - Beginning worker shutdown, will wait 0:00:00 before cancelling activities

Traceback (most recent call last):
  File "/usr/local/lib64/python3.9/site-packages/temporalio/worker/_workflow.py", line 154, in run
    act = await self._bridge_worker().poll_workflow_activation()
  File "/usr/local/lib64/python3.9/site-packages/temporalio/bridge/worker.py", line 147, in poll_workflow_activation
    await self._ref.poll_workflow_activation()
RuntimeError: Poll failure: Unhandled grpc error when polling: Status { code: Unimplemented, metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Wed, 19 Mar 2025 02:17:48 GMT"} }, source: None }

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/temporal/temporal_sample_worker_python/src/run_worker.py", line 202, in <module>
    asyncio.run(main())
  File "/usr/lib64/python3.9/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/lib64/python3.9/asyncio/base_events.py", line 647, in run_until_complete
    return future.result()
  File "/home/temporal/temporal_sample_worker_python/src/run_worker.py", line 179, in main
    await run_worker(client, taskqueue_name, args.worker_type)
  File "/home/temporal/temporal_sample_worker_python/src/run_worker.py", line 142, in run_worker
    await worker.run()
  File "/usr/local/lib64/python3.9/site-packages/temporalio/worker/_worker.py", line 571, in run
    raise exception
  File "/usr/local/lib64/python3.9/site-packages/temporalio/worker/_workflow.py", line 164, in run
    raise RuntimeError("Workflow worker failed") from err
RuntimeError: Workflow worker failed

Environment/Versions

• Temporal Version: Server v1.26.2, SDK v1.10.0

Additional context

Our issue looks similar to this this open bug: [Bug] Worker hangs after polling workflow task queue · Issue #631 · temporalio/sdk-python · GitHub
Wanted to check if it is same issue and if not we need help in understanding the fixing it.