Increase 4MB of gRPC limited in Azure Durable Function for Java

There seems to be a limit only in Azure Durable Function For Java that cannot send payload over 4MB, can you update the limit and align with int.MaxValue in function host runtime? Below are the observation about this issue:

1. Got below error when POST to durable function with a payload over 4MB.

Microsoft.Azure.WebJobs.Host.FunctionInvocationException : Exception while executing function: Functions.StartOrchestration ---> Microsoft.Azure.WebJobs.Script.Workers.Rpc.RpcException : Result: Failure

Exception: StatusRuntimeException: RESOURCE_EXHAUSTED: Received message larger than max (5060857 vs. 4194304)
Stack: java.lang.reflect.InvocationTargetException
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at com.microsoft.azure.functions.worker.broker.JavaMethodInvokeInfo.invoke(JavaMethodInvokeInfo.java:22)
at com.microsoft.azure.functions.worker.broker.EnhancedJavaMethodExecutorImpl.execute(EnhancedJavaMethodExecutorImpl.java:22)
at com.microsoft.azure.functions.worker.chain.FunctionExecutionMiddleware.invoke(FunctionExecutionMiddleware.java:19)
at com.microsoft.azure.functions.worker.chain.InvocationChain.doNext(InvocationChain.java:21)
at com.microsoft.durabletask.azurefunctions.internal.middleware.OrchestrationMiddleware.invoke(OrchestrationMiddleware.java:29)
at com.microsoft.azure.functions.worker.chain.InvocationChain.doNext(InvocationChain.java:21)
at com.microsoft.azure.functions.worker.broker.JavaFunctionBroker.invokeMethod(JavaFunctionBroker.java:125)
at com.microsoft.azure.functions.worker.handler.InvocationRequestHandler.execute(InvocationRequestHandler.java:34)
at com.microsoft.azure.functions.worker.handler.InvocationRequestHandler.execute(InvocationRequestHandler.java:10)
at com.microsoft.azure.functions.worker.handler.MessageHandler.handle(MessageHandler.java:44)
at com.microsoft.azure.functions.worker.JavaWorkerClient$StreamingMessagePeer.lambda$onNext$0(JavaWorkerClient.java:94)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: io.grpc.StatusRuntimeException: RESOURCE_EXHAUSTED: Received message larger than max (5060857 vs. 4194304)
at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:271)
at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:252)
at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:165)
at com.microsoft.durabletask.implementation.protobuf.TaskHubSidecarServiceGrpc$TaskHubSidecarServiceBlockingStub.startInstance(TaskHubSidecarServiceGrpc.java:919)
at com.microsoft.durabletask.DurableTaskGrpcClient.scheduleNewOrchestrationInstance(DurableTaskGrpcClient.java:114)
at com.microsoft.durabletask.DurableTaskClient.scheduleNewOrchestrationInstance(DurableTaskClient.java:67)
at com.microsoft.durabletask.DurableTaskClient.scheduleNewOrchestrationInstance(DurableTaskClient.java:52)
at com.singpost.wms.uniqlo.function.SplitXMLFunction.startOrchestration(SplitXMLFunction.java:75)
... 20 more

at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at async Microsoft.Azure.WebJobs.Script.Description.WorkerFunctionInvoker.InvokeCore(Object[] parameters,FunctionInvocationContext context) at //src/WebJobs.Script/Description/Workers/WorkerFunctionInvoker.cs : 96
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at async Microsoft.Azure.WebJobs.Script.Description.FunctionInvokerBase.Invoke(Object[] parameters) at //src/WebJobs.Script/Description/FunctionInvokerBase.cs : 82
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at async Microsoft.Azure.WebJobs.Script.Description.FunctionGenerator.Coerce[T](Task1 src) at /_/src/WebJobs.Script/Description/FunctionGenerator.cs : 225 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Microsoft.Azure.WebJobs.Host.Executors.FunctionInvoker2.InvokeAsync[TReflected,TReturnValue](Object instance,Object[] arguments) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionInvoker.cs : 52
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.InvokeWithTimeoutAsync(IFunctionInvoker invoker,ParameterHelper parameterHelper,CancellationTokenSource timeoutTokenSource,CancellationTokenSource functionCancellationTokenSource,Boolean throwOnTimeout,TimeSpan timerInterval,IFunctionInstance instance) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs : 581
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithWatchersAsync(IFunctionInstanceEx instance,ParameterHelper parameterHelper,ILogger logger,CancellationTokenSource functionCancellationTokenSource) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs : 527
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithLoggingAsync(IFunctionInstanceEx instance,FunctionStartedMessage message,FunctionInstanceLogEntry instanceLogEntry,ParameterHelper parameterHelper,ILogger logger,CancellationToken cancellationToken) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs : 306
End of inner exception
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithLoggingAsync(IFunctionInstanceEx instance,FunctionStartedMessage message,FunctionInstanceLogEntry instanceLogEntry,ParameterHelper parameterHelper,ILogger logger,CancellationToken cancellationToken) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs : 352
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at async Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.TryExecuteAsync(IFunctionInstance functionInstance,CancellationToken cancellationToken) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs : 108

2. It is regarding the grpcMaxMessageLength, there are multiple issues related to it. In Function Host Runtime, the limit should be set to int.MaxValue:

https://github.com/Azure/azure-functions-host/issues/2677

DefaultMaxMessageLengthBytes = int.MaxValue;
https://github.com/Azure/azure-functions-host/blob/dev/src/WebJobs.Script/Workers/Rpc/RpcWorkerConstants.cs#L38

3. From logs, we see that grpcMaxMessageLength has already been set to 2147483647. Howeve it does not seem to take place in the durable function.

Starting worker process with FileName:C:\Program Files\Java\microsoft-jdk-11.0.16.1/bin/java WorkingDirectory:C:\home\site\wwwroot Arguments: -XX:+TieredCompilation -XX:TieredStopAtLevel=1 -noverify -Djava.net.preferIPv4Stack=true -jar "C:\Program Files (x86)\SiteExtensions\Functions\4.19.0\workers\java\azure-functions-java-worker.jar" --host 127.0.0.1 --port 50628 --workerId ada93a63-8e10-49ac-ba64-19113424c536 --requestId d5533bd6-b6a2-442d-b394-22c67ae1f6b5 --grpcMaxMessageLength 2147483647

4. This issue can always be reproduced from local with latest function host version: 4.21.1.20667 when send a POST with payload larger than 4MB.

5. This issue cannot be reproduced when use Azure Durable Function for Python. Also, no issues with HTTP trigger in normal Java function app. Hence we believe it is only with Azure Durable Function for Java.

6. There is no such setting for "grpc max message size" in this repo found, it seems to inherit default 4MB limit in gRPC.

首先跟踪 grpcMaxMessageLength，从 Java worker 的启动参数一直到 DurableTaskGrpcClient.scheduleNewOrchestrationInstance 路径，并将其与 Function Host 运行时设置以及此处描述的 Python 行为进行比较。使用超过 4 MB 的 POST payload，在 Function Host 4.21.1.20667 中本地复现；当 Java Durable Function 能够接受更大的消息且不出现 gRPC RESOURCE_EXHAUSTED 错误时，即表示完成。