grpc / grpc/grpc-dotnet

HTTP/2 error code 'PROTOCOL_ERROR' (0x1) in .Net 8

Open
#2,544 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
C#
Stars
4.5k
Forks
836
Avg merge
6d 3h
Merged PRs (30d)
7

Description

I have migrated Azure function app from .net 6 to .net8. function app includes orchestrator trigger and activity triggers.and when I am running function app, when calling orchestartor trigger from function its throwing error like below.
### What version of gRPC and what language are you using?

### What operating system (Linux, Windows,...) and version?
Windows

### What runtime / compiler are you using (e.g. .NET Core SDK version `dotnet --info`)
.Net 8

### What did you do?
If possible, provide a recipe for reproducing the error. Try being specific and include code snippets if helpful.

### What did you expect to see?
Azure function should run without errors

### What did you see instead?
[2024-09-25T10:07:16.124Z] Function 'AZF_Amazon', Invocation id 'b422c9b2-a6f8-4d21-abdc-029bc676e834': An exception was thrown by the invocation.
[2024-09-25T10:07:16.131Z] Result: Function 'AZF_Amazon', Invocation id 'b422c9b2-a6f8-4d21-abdc-029bc676e834': An exception was thrown by the invocation.
Exception: System.AggregateException: One or more errors occurred. (Status(StatusCode="Internal", Detail="Error starting gRPC call. HttpRequestException: The HTTP/2 server sent invalid data on the connection. HTTP/2 error code 'PROTOCOL_ERROR' (0x1). (HttpProtocolError) HttpProtocolException: The HTTP/2 server sent invalid data on the connection. HTTP/2 error code 'PROTOCOL_ERROR' (0x1). (HttpProtocolError)", DebugException="System.Net.Http.HttpRequestException: The HTTP/2 server sent invalid data on the connection. HTTP/2 error code 'PROTOCOL_ERROR' (0x1). (HttpProtocolError)"))
[2024-09-25T10:07:16.135Z] ---> Grpc.Core.RpcException: Status(StatusCode="Internal", Detail="Error starting gRPC call. HttpRequestException: The HTTP/2 server sent invalid data on the connection. HTTP/2 error code 'PROTOCOL_ERROR' (0x1). (HttpProtocolError) HttpProtocolException: The HTTP/2 server sent invalid data on the connection. HTTP/2 error code 'PROTOCOL_ERROR' (0x1). (HttpProtocolError)", DebugException="System.Net.Http.HttpRequestException: The HTTP/2 server sent invalid data on the connection. HTTP/2 error code 'PROTOCOL_ERROR' (0x1). (HttpProtocolError)")
[2024-09-25T10:07:16.139Z] ---> System.Net.Http.HttpRequestException: The HTTP/2 server sent invalid data on the connection. HTTP/2 error code 'PROTOCOL_ERROR' (0x1). (HttpProtocolError)
[2024-09-25T10:07:16.141Z] ---> System.Net.Http.HttpProtocolException: The HTTP/2 server sent invalid data on the connection. HTTP/2 error code 'PROTOCOL_ERROR' (0x1). (HttpProtocolError)
[2024-09-25T10:07:16.153Z] at System.Net.Http.Http2Connection.ThrowRequestAborted(Exception innerException)
[2024-09-25T10:07:16.155Z] at System.Net.Http.Http2Connection.Http2Stream.CheckResponseBodyState()
[2024-09-25T10:07:16.156Z] at System.Net.Http.Http2Connection.Http2Stream.TryEnsureHeaders()
[2024-09-25T10:07:16.182Z] at System.Net.Http.Http2Connection.Http2Stream.ReadResponseHeadersAsync(CancellationToken cancellationToken)
[2024-09-25T10:07:16.185Z] at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
[2024-09-25T10:07:16.188Z] --- End of inner exception stack trace ---
[2024-09-25T10:07:16.203Z] at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
[2024-09-25T10:07:16.209Z] at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
[2024-09-25T10:07:16.212Z] at System.Net.Http.DiagnosticsHandler.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
[2024-09-25T10:07:16.215Z] at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
[2024-09-25T10:07:16.217Z] at System.Net.Http.HttpClient.g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
[2024-09-25T10:07:16.219Z] at Grpc.Net.Client.Internal.GrpcCall`2.RunCall(HttpRequestMessage request, Nullable`1 timeout)
[2024-09-25T10:07:16.228Z] --- End of inner exception stack trace ---
[2024-09-25T10:07:16.230Z] at Grpc.Net.Client.Internal.GrpcCall`2.GetResponseHeadersCoreAsync()
[2024-09-25T10:07:16.232Z] at Microsoft.DurableTask.Client.Grpc.GrpcDurableTaskClient.ScheduleNewOrchestrationInstanceAsync(TaskName orchestratorName, Object input, StartOrchestrationOptions options, CancellationToken cancellation)
[2024-09-25T10:07:16.234Z] at PII.ERP.Finance.AccountsReceivable.Remittances.AZF.Functions.Amazon.Triggers.AZF_Amazon.Run(TimerInfo myTimer, DurableTaskClient starter, FunctionContext context) in C:\Users\stulluri\source\repos\AZF.PII.ERP.Finance.AccountsReceivable.Remittances\Azf\Functions\Amazon\Triggers\AZF_Amazon.cs:line 24
[2024-09-25T10:07:16.240Z] at Microsoft.Azure.Functions.Worker.Invocation.VoidTaskMethodInvoker`2.InvokeAsync(TReflected instance, Object[] arguments) in D:\a\_work\1\s\src\DotNetWorker.Core\Invocation\VoidTaskMethodInvoker.cs:line 22
[2024-09-25T10:07:16.243Z] --- End of inner exception stack trace ---
[2024-09-25T10:07:16.244Z] at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
[2024-09-25T10:07:16.248Z] at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
[2024-09-25T10:07:16.250Z] at Microsoft.Azure.Functions.Worker.Invocation.DefaultFunctionInvoker`2.<>c.b__6_0(Task`1 t) in D:\a\_work\1\s\src\DotNetWorker.Core\Invocation\DefaultFunctionInvoker.cs:line 32
[2024-09-25T10:07:16.256Z] at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()
[2024-09-25T10:07:16.258Z] at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
[2024-09-25T10:07:16.261Z] --- End of stack trace from previous location ---
[2024-09-25T10:07:16.262Z] at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
[2024-09-25T10:07:16.265Z] at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
[2024-09-25T10:07:16.271Z] --- End of stack trace from previous location ---
[2024-09-25T10:07:16.273Z] at Microsoft.Azure.Functions.Worker.Invocation.DefaultFunctionExecutor.ExecuteAsync(FunctionContext context) in D:\a\_work\1\s\src\DotNetWorker.Core\Invocation\DefaultFunctionExecutor.cs:line 49
[2024-09-25T10:07:16.276Z] at Microsoft.Azure.Functions.Worker.OutputBindings.OutputBindingsMiddleware.Invoke(FunctionContext context, FunctionExecutionDelegate next) in D:\a\_work\1\s\src\DotNetWorker.Core\OutputBindings\OutputBindingsMiddleware.cs:line 13
[2024-09-25T10:07:16.278Z] at Microsoft.Azure.Functions.Worker.FunctionsApplication.InvokeFunctionAsync(FunctionContext context) in D:\a\_work\1\s\src\DotNetWorker.Core\FunctionsApplication.cs:line 89
Stack: at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
[2024-09-25T10:07:16.280Z] at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
[2024-09-25T10:07:16.282Z] at Microsoft.Azure.Functions.Worker.Invocation.DefaultFunctionInvoker`2.<>c.b__6_0(Task`1 t) in D:\a\_work\1\s\src\DotNetWorker.Core\Invocation\DefaultFunctionInvoker.cs:line 32
[2024-09-25T10:07:16.294Z] at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()
[2024-09-25T10:07:16.296Z] at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
[2024-09-25T10:07:16.313Z] --- End of stack trace from previous location ---
[2024-09-25T10:07:16.315Z] at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
[2024-09-25T10:07:16.324Z] at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
[2024-09-25T10:07:16.326Z] --- End of stack trace from previous location ---
[2024-09-25T10:07:16.328Z] at Microsoft.Azure.Functions.Worker.Invocation.DefaultFunctionExecutor.ExecuteAsync(FunctionContext context) in D:\a\_work\1\s\src\DotNetWorker.Core\Invocation\DefaultFunctionExecutor.cs:line 49
[2024-09-25T10:07:16.330Z] at Microsoft.Azure.Functions.Worker.OutputBindings.OutputBindingsMiddleware.Invoke(FunctionContext context, FunctionExecutionDelegate next) in D:\a\_work\1\s\src\DotNetWorker.Core\OutputBindings\OutputBindingsMiddleware.cs:line 13
[2024-09-25T10:07:16.355Z] at Microsoft.Azure.Functions.Worker.FunctionsApplication.InvokeFunctionAsync(FunctionContext context) in D:\a\_work\1\s\src\DotNetWorker.Core\FunctionsApplication.cs:line 89.
[2024-09-25T10:07:16.451Z] Executed 'Functions.AZF_Amazon' (Failed, Id=b422c9b2-a6f8-4d21-abdc-029bc676e834, Duration=25235ms)
[2024-09-25T10:07:16.673Z] System.Private.CoreLib: Exception while executing function: Functions.AZF_Amazon. System.Private.CoreLib: Result: Failure
Exception: System.AggregateException: One or more errors occurred. (Status(StatusCode="Internal", Detail="Error starting gRPC call. HttpRequestException: The HTTP/2 server sent invalid data on the connection. HTTP/2 error code 'PROTOCOL_ERROR' (0x1). (HttpProtocolError) HttpProtocolException: The HTTP/2 server sent invalid data on the connection. HTTP/2 error code 'PROTOCOL_ERROR' (0x1). (HttpProtocolError)", DebugException="System.Net.Http.HttpRequestException: The HTTP/2 server sent invalid data on the connection. HTTP/2 error code 'PROTOCOL_ERROR' (0x1). (HttpProtocolError)"))
[2024-09-25T10:07:16.682Z] ---> Grpc.Core.RpcException: Status(StatusCode="Internal", Detail="Error starting gRPC call. HttpRequestException: The HTTP/2 server sent invalid data on the connection. HTTP/2 error code 'PROTOCOL_ERROR' (0x1). (HttpProtocolError) HttpProtocolException: The HTTP/2 server sent invalid data on the connection. HTTP/2 error code 'PROTOCOL_ERROR' (0x1). (HttpProtocolError)", DebugException="System.Net.Http.HttpRequestException: The HTTP/2 server sent invalid data on the connection. HTTP/2 error code 'PROTOCOL_ERROR' (0x1). (HttpProtocolError)")
[2024-09-25T10:07:16.686Z] ---> System.Net.Http.HttpRequestException: The HTTP/2 server sent invalid data on the connection. HTTP/2 error code 'PROTOCOL_ERROR' (0x1). (HttpProtocolError)
[2024-09-25T10:07:16.689Z] ---> System.Net.Http.HttpProtocolException: The HTTP/2 server sent invalid data on the connection. HTTP/2 error code 'PROTOCOL_ERROR' (0x1). (HttpProtocolError)
[2024-09-25T10:07:16.702Z] at System.Net.Http.Http2Connection.ThrowRequestAborted(Exception innerException)
[2024-09-25T10:07:16.705Z] at System.Net.Http.Http2Connection.Http2Stream.CheckResponseBodyState()
[2024-09-25T10:07:16.706Z] at System.Net.Http.Http2Connection.Http2Stream.TryEnsureHeaders()
[2024-09-25T10:07:16.708Z] at System.Net.Http.Http2Connection.Http2Stream.ReadResponseHeadersAsync(CancellationToken cancellationToken)
[2024-09-25T10:07:16.722Z] at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
[2024-09-25T10:07:16.733Z] --- End of inner exception stack trace ---
[2024-09-25T10:07:16.735Z] at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
[2024-09-25T10:07:16.749Z] at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
[2024-09-25T10:07:16.752Z] at System.Net.Http.DiagnosticsHandler.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
[2024-09-25T10:07:16.767Z] at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
[2024-09-25T10:07:16.769Z] at System.Net.Http.HttpClient.g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
[2024-09-25T10:07:16.776Z] at Grpc.Net.Client.Internal.GrpcCall`2.RunCall(HttpRequestMessage request, Nullable`1 timeout)
[2024-09-25T10:07:16.779Z] --- End of inner exception stack trace ---
[2024-09-25T10:07:16.780Z] at Grpc.Net.Client.Internal.GrpcCall`2.GetResponseHeadersCoreAsync()
[2024-09-25T10:07:16.782Z] at Microsoft.DurableTask.Client.Grpc.GrpcDurableTaskClient.ScheduleNewOrchestrationInstanceAsync(TaskName orchestratorName, Object input, StartOrchestrationOptions options, CancellationToken cancellation)
[2024-09-25T10:07:16.785Z] at PII.ERP.Finance.AccountsReceivable.Remittances.AZF.Functions.Amazon.Triggers.AZF_Amazon.Run(TimerInfo myTimer, DurableTaskClient starter, FunctionContext context) in C:\Users\stulluri\source\repos\AZF.PII.ERP.Finance.AccountsReceivable.Remittances\Azf\Functions\Amazon\Triggers\AZF_Amazon.cs:line 24
[2024-09-25T10:07:16.793Z] at Microsoft.Azure.Functions.Worker.Invocation.VoidTaskMethodInvoker`2.InvokeAsync(TReflected instance, Object[] arguments) in D:\a\_work\1\s\src\DotNetWorker.Core\Invocation\VoidTaskMethodInvoker.cs:line 22
[2024-09-25T10:07:16.795Z] --- End of inner exception stack trace ---
[2024-09-25T10:07:16.797Z] at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
[2024-09-25T10:07:16.809Z] at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
[2024-09-25T10:07:16.810Z] at Microsoft.Azure.Functions.Worker.Invocation.DefaultFunctionInvoker`2.<>c.b__6_0(Task`1 t) in D:\a\_work\1\s\src\DotNetWorker.Core\Invocation\DefaultFunctionInvoker.cs:line 32
[2024-09-25T10:07:16.813Z] at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()
[2024-09-25T10:07:16.814Z] at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
[2024-09-25T10:07:16.816Z] --- End of stack trace from previous location ---
[2024-09-25T10:07:16.821Z] at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
[2024-09-25T10:07:16.824Z] at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
[2024-09-25T10:07:16.826Z] --- End of stack trace from previous location ---
[2024-09-25T10:07:16.827Z] at Microsoft.Azure.Functions.Worker.Invocation.DefaultFunctionExecutor.ExecuteAsync(FunctionContext context) in D:\a\_work\1\s\src\DotNetWorker.Core\Invocation\DefaultFunctionExecutor.cs:line 49
[2024-09-25T10:07:16.829Z] at Microsoft.Azure.Functions.Worker.OutputBindings.OutputBindingsMiddleware.Invoke(FunctionContext context, FunctionExecutionDelegate next) in D:\a\_work\1\s\src\DotNetWorker.Core\OutputBindings\OutputBindingsMiddleware.cs:line 13
[2024-09-25T10:07:16.831Z] at Microsoft.Azure.Functions.Worker.FunctionsApplication.InvokeFunctionAsync(FunctionContext context) in D:\a\_work\1\s\src\DotNetWorker.Core\FunctionsApplication.cs:line 89
[2024-09-25T10:07:16.842Z] at Microsoft.Azure.Functions.Worker.Handlers.InvocationHandler.InvokeAsync(InvocationRequest request) in D:\a\_work\1\s\src\DotNetWorker.Grpc\Handlers\InvocationHandler.cs:line 88
Stack: at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
[2024-09-25T10:07:16.844Z] at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
[2024-09-25T10:07:16.845Z] at Microsoft.Azure.Functions.Worker.Invocation.DefaultFunctionInvoker`2.<>c.b__6_0(Task`1 t) in D:\a\_work\1\s\src\DotNetWorker.Core\Invocation\DefaultFunctionInvoker.cs:line 32
[2024-09-25T10:07:16.847Z] at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()
[2024-09-25T10:07:16.854Z] at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
[2024-09-25T10:07:16.856Z] --- End of stack trace from previous location ---
[2024-09-25T10:07:16.857Z] at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
[2024-09-25T10:07:16.859Z] at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
[2024-09-25T10:07:16.861Z] --- End of stack trace from previous location ---
[2024-09-25T10:07:16.863Z] at Microsoft.Azure.Functions.Worker.Invocation.DefaultFunctionExecutor.ExecuteAsync(FunctionContext context) in D:\a\_work\1\s\src\DotNetWorker.Core\Invocation\DefaultFunctionExecutor.cs:line 49
[2024-09-25T10:07:16.878Z] at Microsoft.Azure.Functions.Worker.OutputBindings.OutputBindingsMiddleware.Invoke(FunctionContext context, FunctionExecutionDelegate next) in D:\a\_work\1\s\src\DotNetWorker.Core\OutputBindings\OutputBindingsMiddleware.cs:line 13
[2024-09-25T10:07:16.883Z] at Microsoft.Azure.Functions.Worker.FunctionsApplication.InvokeFunctionAsync(FunctionContext context) in D:\a\_work\1\s\src\DotNetWorker.Core\FunctionsApplication.cs:line 89
[2024-09-25T10:07:16.885Z] at Microsoft.Azure.Functions.Worker.Handlers.InvocationHandler.InvokeAsync(InvocationRequest request) in D:\a\_work\1\s\src\DotNetWorker.Grpc\Handlers\InvocationHandler.cs:line 88.

Make sure you include information that can help us debug (full error message, exception listing, stack trace, logs).

See [TROUBLESHOOTING.md](https://github.com/grpc/grpc-dotnet/blob/master/TROUBLESHOOTING.md) for how to diagnose problems better.

### Anything else we should know about your project / environment?

Contributor guide

Open the contributing guide

Research direction

Start with the failing call at AZF_Amazon.cs line 24 and trace it through Microsoft.Azure Functions DurableTaskClient and Grpc.Net.Client.Internal. Reproduce the .NET 8 Windows Azure Functions scenario and inspect the HTTP/2 response that produces PROTOCOL_ERROR. Done means identifying a reproducible cause and confirming that orchestration-trigger calls complete without the protocol error.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, csharp, grpc
Domain
api, backend, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.