dotnet / dotnet/yarp

Handle destination gray failures better (Degraded health state)

Open
#2,071 1 comment 2 reactions 1 assignee Claimed by @benjaminpetit View on GitHub
Type: Idea
Dominant language
C#
Stars
9.6k
Forks
933
Avg merge
12d 18h
Merged PRs (30d)
2

Description

### What should we add or change to make your life better?

The existing health policies (e.g. ConsecutiveFailuresHealthPolicy) can be problematic when destinations are not entirely up nor entirely down. This can happen in practice, and the current policies produce sub-optimal routing decisions.

### Why is this important to you?
This would help maintain high availability for external scenarios despite transient instability of internal services and/or machines.

### Proposal
* Introduce a new `Degraded` destination health state (see also: #1011)
* Modify destination selection policy to prefer destinations in Healthy state rather than Destination, but still pick Degraded destinations if there are "few" (according to some criteria) Healthy ones
* Modify active health checks to allow a destination to declare itself to be degraded if it would like to not be a preferred choice for traffic, while allowing YARP to make the final decision whether to use this destination or not based on all signals it has access to
* Destination health transitions should avoid spurious Health and Unhealth determinations based on single health observations. The diagram below proposes a symmetric decision process where a transition from Degraded state requires three consecutive consistent health observations

```mermaid
---
title: Destination health state diagram (PROPOSAL)
---

stateDiagram-v2
[*] --> Unknown
Unknown --> Healthy: active probe == "success"
Unknown --> Unhealthy: active probe != "success"
Degraded --> Healthy: active probe == "success"\n(3rd consecutive)
Healthy --> Degraded: proxy result == "degraded" OR\nactive probe == "degraded" OR\nactive probe !="success" in 2 of past 5 evals
Degraded --> Unhealthy: active probe != "success"\n(3rd consecutive)
Unhealthy --> Degraded: active probe =="success"
```

NOTE: This is related to #1011 but goes beyond it in scope. Filing as a separate issue seemed appropriate.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.