[DISCUSSION] Circuit breaker will remain half-open state forever when the request is blocked by upcoming rules
- Dominant language
- Java
- Stars
- 23.1k
- Forks
- 8.1k
- PR merge metrics
- No merged PRs in 30d
Description
## Issue Description
Type: *bug report*
### Describe what happened (or what feature you want)
The circuit breaker won't recover from half-open state when the request is actually blocked by upcoming rules. For example, there are two degrade rules **of the same resource**: R1(circuit breaker state=OPEN, recoveryTimeout=10s) and R2(circuit breaker state=OPEN, recoveryTimeout=20s)
There may be circumstances when R1 has reached the recovery timepoint but R2 has not. If a request comes, there will be transformation: **R1(OPEN → HALF-OPEN), R2(OPEN)**
This request will be allowed by R1 but rejected by R2, thus finally blocked. The invocation won't actually occur, so it will **NEVER** complete. For state transformation from HALF-OPEN to OPEN/CLOSED, it should happen only when invocation completes, so for R1 the associated circuit breaker state will be HALF-OPEN forever. Actually this could happen when there are any *blocked rules* after R1 (not only degrade rules).
This is a fatal bug and should be carefully resolved. We may need a temporary workaround for the half-open case in 1.8.0, then improve the overall design later. Discussions are welcomed.
Original discussions can be found in https://github.com/alibaba/Sentinel/pull/1490#discussion_r462079959
Contributor guide
Research direction
Start with the original discussion in pull request #1490 and trace the circuit-breaker state transitions for two rules on the same resource. Reproduce the case where the first rule becomes HALF-OPEN but a later rule blocks the request; done means the circuit breaker cannot remain HALF-OPEN forever when invocation never occurs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100