Azure / Azure/azure-relay-dotnet

High Request Volume Causes Timeout in Hybrid Connection Listener

Open
#178 0 comments 1 reaction 1 assignee Claimed by @jbjordan View on GitHub
customer-reported
Dominant language
C#
Stars
41
Forks
43
PR merge metrics
No merged PRs in 30d

Description

## Description

When the Hybrid Connection Listene rexperiences a high volume of requests in a short period (e.g., 1,000 requests within a minute), it encounters a timeout exception.

## Root Cause:

The issue seems to stem from how the` HybridHttpConnection` class in the Microsoft.Azure.Relay package handles incoming requests. Specifically, in the `CreateAsync `method, `Task.Run` is used to spawn a new task for each request.

During a period of request spikes, if the request handler takes a long (eg. 100ms) time to execute, this will quickly consume the readily available threads from the ThreadPool. A significant delay is thereafter introduced for the creation of each new thread, leading
to increasignly long response times.

Further adding to the problem is that Azure Relay itself has a maximum timeout of 60 seconds, causing the request to fail if a response has not been received in that time.

Normally a long running request handler in ASP.NET Core is created as an async method allowing I/O operations to not block the threads. However, `HybridConnectionClient.RequestHandler` is an `Action` preventing us from using async techniques to alleviate this problem.

Suggestion is to make `HybridConnectionListener.RequestHandler` async, and `await` it from within `ProcessFirstRequestAsync`. This would be a breaking change, so perhaps an alternative would be to introduce a new property `Func RequestHandlerAsync` that is used if it is set instead of the current `RequestHandler`

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.