dotnet / dotnet/yarp

Sudden spike in TCP Close Wait

Open
#2,837 2 comments 0 reactions 0 assignees View on GitHub
Type: Bug
Dominant language
C#
Stars
9.6k
Forks
933
Avg merge
12d 18h
Merged PRs (30d)
2

Description

### Describe the bug
We started seeing a spike in TCP Close Wait / Http Queue Length (+4k) on one of our Azure App Services for Windows running YARP. Scaling out to one more instance (2->3) fixed the issue temporarily, after scaling down we had another spike. This is the first occurence of this issue in the last ~4 years we've been running YARP. Underlying infrastructure or code hasn't been changed in the last 2 months.

### Further technical details
- Azure App Service on Windows (P2v3), VNET Integration / PaaS access over Private Endpoints. Some calls are made to a AFD load-balanced service (Eg. public endpoint which are subject to SNAT restrictions)
- YARP 2.1, default HttpClient settings
- The AFD load-balanced service is reached with a typed HttpClient using a handler lifetime of 90 seconds (Down from 2 minutes)
- NET8
- YARP forwards requests to ~20 clusters (Unique destinations, also Azure App Services) with traffic being about 150 requests/s in total
- Requests on a few specific endpoints may take over 5-10 seconds, but those are limited.

Could there be a HttpClient setting we can optizime to release / close HttpClients earlier?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.