alibaba / alibaba/Sentinel

Under concurrent traffic, (possibly) 1. Current limit is not allowed. 2. All the requests are blocked. (Under concurrent traffic, rate limit may not correct, and block all request)

Open
#957 3 comments 0 reactions 0 assignees View on GitHub
kind/question
Dominant language
Java
Stars
23.1k
Forks
8.1k
PR merge metrics
No merged PRs in 30d

Description

## 并发流量下,(可能)1.限流不准/2.拦截所有请求
## Under concurrent traffic,rate limit may not correct, and block all request

### 并发流量, 限流可能不准(放过更多的流量,稳定复现)
中文:
在较大的流量压力(并发)下。限流**可能**不准确。 不准确的点在于, 获取`entry`责任链中, `StatisticSlot` 统计数据, 与`FlowSlot`获取数据,他们两并不是抢占式的。在我的电脑(i5 8400, 16G, MAC10.14.5)上,他们两大约有2ms左右的时间差,而这个时间差足够让后续的线程绕过限流`FlowSlot`插件了。

English:
Under concurrent traffic, rate limit may not correct;
the way for limit is a way of slot chain:
- the slot `FlowSlot` for limiting,
- the slot `StatisticSlot` for statistic;

`FlowSlot` use `StatisticSlot` data. but they do not wait for a lock.
it may cause `FlowSlot` use an old data. And pass a lot of traffic.

### 并发流量下, 部分的请求可能都会被block掉(通过的流量低于设定阈值, 仅出现在线程池调度的情况下)。
中文:
仅在线程池环境下, 该问题复现, 且稳定复现。 我看了很久,没有判断出原因是什么。
现状: 可能在持续长达5s以上的时间, 有流量进入,限流大小不小于1,所有请求被阻断。
English:
only under thread pool, the request may be blocked by sentinel, all the request be blocked, even the limit is larger than 0.

> 这两个问题一般一起出现, 先出现第一个问题, 大约两三秒之后出现第二个问题。 第一个问题导致放过更多的流量, 第二个问题可能导致所有流量全部被block(第二个问题只出现在线程池环境下)。
> the 2 case show together; the first one may cause more than limit traffic; the second may cause no traffic passed(the second case may only appear in thread pool );

### 复现方式(the way to reproduce it)

1. 在主线程中不停的创建子线程, 子线程采用Sentinel推荐的写法(模拟tomcat环境)
(create a lot of child thread, only like this)
```
public static void main(String[] xxx) throws Exception {
XXX x = new XXX();
int j = 99999999;
while (j-- > 0) {
try {
final int av = j;
executor.execute(() -> x.打印一个SystemOut(av));
} catch (Exception E) {
E.printStackTrace();
}
// 当执行休眠的时候, 问题不再复现
/*TimeUnit.MILLISECONDS.sleep(20L);*/
}
System.out.println("shut down");
}
```
2. 使用Sentinel推荐的编码方式
(each thread call the method, like sentinel told us)
3. 在程序运行时, 开启限流。
(open limit on sentinel-dashboard)

### Tell us your environment
mac os x 10.14.5, jdk8u225, eclipse

### Anything else we need to know?
在 限流器 获取当前QPS的时候, 同步一下, 可解决这两个问题。

## 复现问题的全部代码:
```
import java.text.SimpleDateFormat;
import java.util.ArrayList;
import java.util.Date;
import java.util.List;
import java.util.concurrent.Executor;
import java.util.concurrent.Executors;

import com.alibaba.csp.sentinel.Entry;
import com.alibaba.csp.sentinel.SphU;
import com.alibaba.csp.sentinel.slots.block.RuleConstant;
import com.alibaba.csp.sentinel.slots.block.flow.FlowRule;
import com.alibaba.csp.sentinel.slots.block.flow.FlowRuleManager;

public class ZZZ {

SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");

private static Executor executor = Executors.newFixedThreadPool(50);

private static String pre;

private static int size = 99990000;

public void justPrint(int i) {
try (Entry entry = SphU.entry("XXX.sysGood11")) {
String print = sdf.format(new Date());
if (!print.equalsIgnoreCase(pre)) {
System.out.println();
pre = print;
}
System.out.println(print + "\t" + i);
} catch (Throwable e) {
} finally {
}
}

public static void main(String[] xxx) throws Exception {
ZZZ x = new ZZZ();
int j = size;

initFlowQpsRule();

while (j-- > 0) {
try {
final int p = j;
executor.execute(() -> x.justPrint(p));
/*new Thread(() -> x.justPrint(p)).start()*/
} catch (Exception E) {
E.printStackTrace();
}
/*TimeUnit.MILLISECONDS.sleep(20L);*/
}
System.out.println("shut down");
}

private static void initFlowQpsRule() {
List rules = new ArrayList<>();
FlowRule rule = new FlowRule("XXX.sysGood11");
// set limit qps to 5
rule.setCount(5);
rule.setGrade(RuleConstant.FLOW_GRADE_QPS);
rule.setLimitApp("default");
rules.add(rule);
FlowRuleManager.loadRules(rules);
}
}

```

Contributor guide

Open the contributing guide

Research direction

Start with the reported reproducer using SphU.entry, a fixed thread pool, FlowRuleManager, FlowSlot, and StatisticSlot. Reproduce the QPS limit of 5 under concurrent traffic, then trace how FlowSlot reads StatisticSlot data; done means the concurrency case no longer over-admits traffic or blocks all requests unexpectedly.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.