Azure / Azure/azure-functions-host

ReadToEnd() throws exception on slow requests; introduce MinRequestBodyDataRate

Open
#3,866 4 comments 4 reactions 0 assignees View on GitHub
Dominant language
C#
Stars
2k
Forks
482
Avg merge
2d 10h
Merged PRs (30d)
36

Description

Some http requests from our client are sent to the Azure Function very slowly. This causes the Azure Function to throw the following exception:

```
Microsoft.Azure.WebJobs.Host.FunctionInvocationException : Exception while executing function: ReceiveDataHttpTrigger ---> Microsoft.AspNetCore.Connections.ConnectionResetException : The client has disconnected ---> System.IO.EndOfStreamException : Attempted to read past the end of the stream.
```

It would be great if the Azure Function Host has the flexibility to specify the ```MinRequestBodyDataRate```. This was the solution for a WebApi deployment as you see below.

#### Investigative information

- Function App version (1.0 or 2.0): 2.0
- Function name(s) (as appropriate): ReceiveDataHttpTrigger
- Invocation ID: 7e8ac8eb-fbee-408d-bbb4-597d45b56355
- Region: North Central US

##### Exception

```
Microsoft.Azure.WebJobs.Host.FunctionInvocationException : Exception while executing function: ReceiveDataHttpTrigger ---> Microsoft.AspNetCore.Connections.ConnectionResetException : The client has disconnected ---> System.IO.EndOfStreamException : Attempted to read past the end of the stream.
End of inner exception
at Microsoft.AspNetCore.Server.IIS.Core.IO.AsyncIOOperation.GetResult(Int16 token)
at async Microsoft.AspNetCore.Server.IIS.Core.IISHttpContext.ReadBody()
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.IO.Pipelines.PipeCompletion.ThrowLatchedException()
at System.IO.Pipelines.Pipe.GetReadResult(ReadResult& result)
at System.IO.Pipelines.Pipe.GetReadAsyncResult()
at System.IO.Pipelines.Pipe.DefaultPipeReader.GetResult(Int16 token)
at async Microsoft.AspNetCore.Server.IIS.Core.IISHttpContext.ReadAsync(Memory`1 memory,CancellationToken cancellationToken)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at async Microsoft.AspNetCore.Server.IIS.Core.HttpRequestStream.ReadAsyncInternal(Memory`1 buffer,CancellationToken cancellationToken)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at Microsoft.AspNetCore.Server.IIS.Core.HttpRequestStream.Read(Byte[] buffer,Int32 offset,Int32 count)
at Microsoft.AspNetCore.Server.IIS.Core.WrappingStream.Read(Byte[] buffer,Int32 offset,Int32 count)
at Microsoft.AspNetCore.WebUtilities.FileBufferingReadStream.Read(Byte[] buffer,Int32 offset,Int32 count)
at System.IO.StreamReader.ReadBuffer()
at System.IO.StreamReader.ReadToEnd()
```

#### Repro steps

Unfortunately we are unable to reproduce the problem. The client that interfaces with our Azure Function sends random requests at a very slow speed. This causes the ```System.IO.StreamReader.ReadToEnd()``` to fail on the request body.

What we investigated is that the same code deployed as a WebApi throws an error on the same```System.IO.StreamReder.ReadToEnd()```. However it provides us with some more detail.

```
Microsoft.AspNetCore.Server.Kestrel.Core.BadHttpRequestException: Reading the request body timed out due to data arriving too slowly. See MinRequestBodyDataRate.
at Microsoft.AspNetCore.Server.Kestrel.Core.BadHttpRequestException.Throw(RequestRejectionReason reason)
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Http1MessageBody.PumpAsync()
at System.IO.Pipelines.PipeCompletion.ThrowLatchedException()
at System.IO.Pipelines.Pipe.GetReadResult(ReadResult& result)
at System.IO.Pipelines.Pipe.GetReadAsyncResult()
at System.IO.Pipelines.Pipe.DefaultPipeReader.GetResult(Int16 token)
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.MessageBody.ReadAsync(Memory`1 buffer, CancellationToken cancellationToken)
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpRequestStream.ReadAsyncInternal(Memory`1 buffer, CancellationToken cancellationToken)
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpRequestStream.Read(Byte[] buffer, Int32 offset, Int32 count) at System.IO.StreamReader.ReadBuffer()
at System.IO.StreamReader.ReadToEnd()
```

#### Expected behavior

As with the other requests, we would expect the req.Body to be read correctly even if the request is sent very slowly.

#### Actual behavior

System.IO.StreamReader.ReadToEnd() throws an exception when http request is performed to slow.

#### Known workarounds

When we publish the code as a WebApi we have the option to disable the request data rate with the MinRequestBodyDataRate parameter. Since we applied this parameter no occurrences of the

```
.UseKestrel(options =>
{
options.Limits.MinRequestBodyDataRate = null;
})
```

It would be great if the Azure Function Hosts has a similar setting that would allow us to disable, set a different value than the default 240 bytes/sec.

#### Related information

Provide any related information

* C#

Source
The source below is simplified to show you the call that we make to read the request body. We encapsulated it into some custom methods/classes, etc.

```csharp
[FunctionName("ReceiveDataHttpTrigger")]
public static async Task RunAsync([HttpTrigger(AuthorizationLevel.Function, "post", Route = "data")]HttpRequest req, ExecutionContext context, ILogger log)
{
log.LogDebug("Request received for path={path}", req.Path.ToString());

var body = new StreamReader(req.Body, System.Text.Encoding.UTF8).ReadToEnd();

return new OkResult();
}
```

Contributor guide

Open the contributing guide

Research direction

Start with the Azure Functions host's HTTP startup path and the existing Kestrel UseKestrel options, then trace how ReceiveDataHttpTrigger reads req.Body. Compare the requested MinRequestBodyDataRate behavior with the shown slow-request exception; done means the host exposes a configurable rate, including disabling it, so slow request bodies are handled as expected.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, csharp
Domain
api, backend, cloud
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.