alibaba / alibaba/Sentinel

高并发情况下,各种规则限流并不准确

Open
#1,620 5 comments 6 reactions 0 assignees View on GitHub
area/flow-control
Dominant language
Java
Stars
23.1k
Forks
8.1k
PR merge metrics
No merged PRs in 30d

Description

先来说下,sentinel-1.6.3源码中提供的demo,测试还是蛮稳定的,让笔者一直以为这个sentinel应该是比较稳定的,直到自己系统准备引入的时候,在高并发情况下压测的时候才发现限流特别的不稳定。

先看一个基本的demo,FlowQpsDemo,笔者简单的修改了一下,如下所示:

`

public void run() {
while (!stop) {
Entry entry = null;

try {
entry = SphU.entry(KEY);
// token acquired, means pass

// 模拟业务操作,50Ms以内的业务操作
Random random2 = new Random();
try {
TimeUnit.MILLISECONDS.sleep(random2.nextInt(50));
} catch (InterruptedException e) {
// ignore
}

pass.addAndGet(1);
} catch (BlockException e1) {
block.incrementAndGet();
} catch (Exception e2) {
// biz exception
} finally {
total.incrementAndGet();
if (entry != null) {
entry.exit();
}
}
`

原本这个50Ms的停顿在finally entry.exit之外,就是在限流动作全部执行完成之后才进行的sleep,这个时候测试是没有问题的,限流很稳定。

但是一旦按照笔者这种改法来测试的话,就会有问题了,测试结果如下:
`

1595158591552, total:833752, pass:50, block:833702
98 send qps is: 1039153
1595158592743, total:1039153, pass:26, block:1039128
97 send qps is: 978964
1595158594133, total:978964, pass:20, block:978942
96 send qps is: 784002
1595158594879, total:784002, pass:25, block:783977

`

按照sentinel提供的使用方式,笔者这种改法无可厚非,但是为什么差距这么大呢?
仔细分析了代码之后发现是限流在并发方面限制的有问题。
以下是DefaultController.canPass()方法:
`

public boolean canPass(Node node, int acquireCount, boolean prioritized) {
int curCount = avgUsedTokens(node);
// 当前qps+1如果大于规则设定的count,则返回false,否则返回true
if (curCount + acquireCount > count) {
...
}
return false;
}
return true;
}

`

avgUsedTokens()方法:
`

private int avgUsedTokens(Node node) {
if (node == null) {
return DEFAULT_AVG_USED_TOKENS;
}
// 获取当前qps或当前线程数
return grade == RuleConstant.FLOW_GRADE_THREAD ? node.curThreadNum() : (int)(node.passQps());
}

`

这段代码的问题在于,如果多个线程(假如有100个)并发执行到 if (curCount + acquireCount > count) ,这里并没有加锁之类的操作,则这100个线程都会返回true,限流失效。

而最终增加node.qps计数的动作在StatisticSlot
`

public void entry(Context context, ResourceWrapper resourceWrapper, DefaultNode node, int count,
boolean prioritized, Object... args) throws Throwable {
try {
// Do some checking.
fireEntry(context, resourceWrapper, node, count, prioritized, args);

// Request passed, add thread count and pass count.
node.increaseThreadNum();
// 这个时候才会增加qps,但是刚才的DefaultController.pass方法已经返回了true,
node.addPassRequest(count);

`

究其原因,还是因为DefaultController在计数时并没有并发限制。

Contributor guide

Open the contributing guide

Research direction

Start with DefaultController.canPass() and its avgUsedTokens() helper, then follow StatisticSlot.entry() to trace when thread and pass counts are updated. Reproduce the modified FlowQpsDemo under concurrent load and compare the observed pass and block counts with the configured limit; done means the behavior is accurate under that scenario.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.