alibaba / alibaba/Sentinel

集群限流出现间歇性抖动

Open
#2,021 0 comments 0 reactions 0 assignees View on GitHub
area/cluster-flow area/performance
Dominant language
Java
Stars
23.1k
Forks
8.1k
PR merge metrics
No merged PRs in 30d

Description

设置超时时间5ms,排除所有的业务,仅仅开启线程压测,使用50个线程,每个线程调用NettyTransportClient#sendRequest方法(请求一个不存在的flowId,从而不计算限流直接返回tokenresult(3),从而达到最快返回的效果),调用后使用sleep(10)睡眠10ms
发现每隔几秒到几十秒会出现抖动,瞬间超时几十次
机器配置2核4G
为了排除网络原因,将client和token server部署在同一台机器上,使用localhost通讯,结果不变,依旧每隔几到几十秒出现瞬间几十次超时
为了排除GC原因,使用G1,使用xms增大初始堆,用arthas观察,
gc.g1_young_generation.count 增长频率明显低于偶发的超时的周期,应该可以排除GC STW导致延迟了
同时尝试增大client端的SO_RCVBUF, SO_SNDBUF,都未能解决问题
cpu负载server端(6%),client端(25%)(未归一化),看上去也不是CPU负载太高导致的

想请问下这个是netty的问题么还是linux上有什么参数需要调优?

Contributor guide

Open the contributing guide

Research direction

Start with NettyTransportClient#sendRequest and reproduce the 5 ms timeout workload using 50 threads, 10 ms sleeps, and a nonexistent flowId. Compare the intermittent timeout bursts with the client/server CPU, GC observations, socket buffer settings, and localhost deployment described in the report. Done means identifying whether the cause is in Netty, Linux tuning, or the client/server behavior and documenting a verified resolution.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems, networking
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.