alibaba / alibaba/Sentinel

[DISCUSSION] Circuit breaker will remain half-open state forever when the request is blocked by upcoming rules

Open
#1,638 8 comments 0 reactions 0 assignees View on GitHub
area/circuit-breaking help wanted kind/bug
Dominant language
Java
Stars
23.1k
Forks
8.1k
PR merge metrics
No merged PRs in 30d

Description

## Issue Description

Type: *bug report*

### Describe what happened (or what feature you want)

The circuit breaker won't recover from half-open state when the request is actually blocked by upcoming rules. For example, there are two degrade rules **of the same resource**: R1(circuit breaker state=OPEN, recoveryTimeout=10s) and R2(circuit breaker state=OPEN, recoveryTimeout=20s)

There may be circumstances when R1 has reached the recovery timepoint but R2 has not. If a request comes, there will be transformation: **R1(OPEN → HALF-OPEN), R2(OPEN)**

This request will be allowed by R1 but rejected by R2, thus finally blocked. The invocation won't actually occur, so it will **NEVER** complete. For state transformation from HALF-OPEN to OPEN/CLOSED, it should happen only when invocation completes, so for R1 the associated circuit breaker state will be HALF-OPEN forever. Actually this could happen when there are any *blocked rules* after R1 (not only degrade rules).

This is a fatal bug and should be carefully resolved. We may need a temporary workaround for the half-open case in 1.8.0, then improve the overall design later. Discussions are welcomed.

Original discussions can be found in https://github.com/alibaba/Sentinel/pull/1490#discussion_r462079959

Contributor guide

Open the contributing guide

Research direction

Start with the original discussion in pull request #1490 and trace the circuit-breaker state transitions for two rules on the same resource. Reproduce the case where the first rule becomes HALF-OPEN but a later rule blocks the request; done means the circuit breaker cannot remain HALF-OPEN forever when invocation never occurs.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.