集群限流出现间歇性抖动
- Dominant language
- Java
- Stars
- 23.1k
- Forks
- 8.1k
- PR merge metrics
- No merged PRs in 30d
Description
设置超时时间5ms,排除所有的业务,仅仅开启线程压测,使用50个线程,每个线程调用NettyTransportClient#sendRequest方法(请求一个不存在的flowId,从而不计算限流直接返回tokenresult(3),从而达到最快返回的效果),调用后使用sleep(10)睡眠10ms
发现每隔几秒到几十秒会出现抖动,瞬间超时几十次
机器配置2核4G
为了排除网络原因,将client和token server部署在同一台机器上,使用localhost通讯,结果不变,依旧每隔几到几十秒出现瞬间几十次超时
为了排除GC原因,使用G1,使用xms增大初始堆,用arthas观察,
gc.g1_young_generation.count 增长频率明显低于偶发的超时的周期,应该可以排除GC STW导致延迟了
同时尝试增大client端的SO_RCVBUF, SO_SNDBUF,都未能解决问题
cpu负载server端(6%),client端(25%)(未归一化),看上去也不是CPU负载太高导致的
想请问下这个是netty的问题么还是linux上有什么参数需要调优?
Contributor guide
Research direction
Start with NettyTransportClient#sendRequest and reproduce the 5 ms timeout workload using 50 threads, 10 ms sleeps, and a nonexistent flowId. Compare the intermittent timeout bursts with the client/server CPU, GC observations, socket buffer settings, and localhost deployment described in the report. Done means identifying whether the cause is in Netty, Linux tuning, or the client/server behavior and documenting a verified resolution.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- distributed-systems, networking
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100