Gracefully fail gRPC-message-too-large issues in worker

Describe the solution you'd like

Also see https://github.com/temporalio/sdk-go/issues/995. Basically this comes back as resource exhausted which is retried by gRPC client, and we should be failing the task.

So we should 1) create a new WorkflowTaskFailedCause for gRPC-message-too-large, and 2) update SDKs to capture the error in clients and convert to non-retryable failure, 3) when failing task because of this failure, clearly specify the enum. The SDK side needs a test confirming the behavior because identifying this error from the server is likely an ugly English-string-check conditional since it is embedded into gRPC server logic. Also, SDKs need to make sure to validate this against cloud which may return a different error due to network/proxy layers.

There is a general server issue open at https://github.com/temporalio/temporal/issues/7777 to make large tasks work, but that is unrelated to this graceful failure.

EDIT: May need to review requirements above, it may be enough to just stop retrying in these situations

Per-SDK Tickets

• API - https://github.com/temporalio/api/pull/591
• Go - https://github.com/temporalio/sdk-go/issues/995
• Java - https://github.com/temporalio/sdk-java/issues/1585
• Core
  • https://github.com/temporalio/sdk-core/issues/462
  • https://github.com/temporalio/sdk-core/issues/970
• TypeScript - N/A
• Python - N/A
• .NET - N/A
• Ruby - N/A
• PHP - N/A
• Temporal CLI - N/A

Start with the linked API pull request and the two open Core tickets, then review the completed and not-applicable per-SDK tickets for the current scope. Done means oversized gRPC task failures are identified, represented by the specified enum, and treated as non-retryable with SDK and cloud behavior validated.