dotnet / dotnet/yarp

Feature request: ability to monitor connection draining

Open
#2,225 1 comment 2 reactions 0 assignees View on GitHub
Type: Idea
Dominant language
C#
Stars
9.6k
Forks
933
Avg merge
12d 18h
Merged PRs (30d)
2

Description

(Follow-up from this PR #2223)

### What should we add or change to make your life better?

When destinations (and entire clusters) are removed from YARP config they will still be used for some period of time: we may be waiting on responses for previously-sent requests, we may even have requests that are about to be sent to a destination but haven't made it out the door yet (e.g. maybe we just picked the outgoing destination in load balancing.)

It would be helpful to be able to monitor this.

We are investigating polling WeakReferences to outgoing DestinationState/ClusterState objects in a background job as a way to implement this on our side. This may work but feels a little fragile/risky.

If we were able to subscribe to events (e.g. if there was an interface for this in Yarp.Telemetry.Consumption) rather than poll that would be convenient.

### Why is this important to you?

At the cluster level, we use clusters to roll out new versions of our software. A new cluster is created with new destinations, and routes are switched to this new cluster. We would like to know if/when a cluster becomes unused so that our deployment tooling can pause before tearing down old resources.

The same thing happens at the destination level but for a different reason typically: our service(s) scale in and out based on load, and when we scale in we need to chose a server to remove from YARP, wait for connections to finish, and then terminate the VM.

In both of these scenarios we'd have an upper bound for requests to finish, this monitoring would enable us to react faster in the (common) happy case where things don't hit the upper bound. That upper bound could be high if you occasionally serve large downloads.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.