apache / apache/pulsar

[BUG] ThresholdShedder Bundles Inaccurate traffic statistics

Open
#16,145 1 comment 0 reactions 0 assignees View on GitHub
Stale type/bug
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

broker DEBUG Log in leader :

2022-06-20 18:50:20.324 [pulsar-load-manager-1-1] INFO o.a.p.broker.loadbalance.impl.ThresholdShedder - [x.x.x.x:8080] broker is planning to shed throughput 0.0 MByte/s less than minimumThroughputThreshold 10.0 MByte/s, skipping bundle unload.

ZK STAT

get /loadbalance/broker-time-average/x.x.x.x:8080

{"shortTermMsgThroughputIn":3.8429246523532754E-20,"shortTermMsgThroughputOut":1.5286027972266306E-23,"shortTermMsgRateIn":8.361056820071499E-23,"shortTermMsgRateOut":2.9078053094894676E-26,"longTermMsgThroughputIn":2.51609898074917E7,"longTermMsgThroughputOut":75116.58169305474,"longTermMsgRateIn":54734.34121373813,"longTermMsgRateOut":34.42857810199835}

Other Server Stats look ok

{"shortTermMsgThroughputIn":1.955625028980416E7,"shortTermMsgThroughputOut":4.911981732451391E-26,"shortTermMsgRateIn":42499.15646885524,"shortTermMsgRateOut":7.995873567619153E-29,"longTermMsgThroughputIn":1.4583922183369141E7,"longTermMsgThroughputOut":2301.2118711286466,"longTermMsgRateIn":31706.082609610632,"longTermMsgRateOut":0.9013642705395061}

BUT
current server CPU of broker useage is 1300+%CPU and I am sure throughput in and out max than 10.0 MByte/s

Contributor guide

Open the contributing guide

Research direction

Start with the broker DEBUG output from ThresholdShedder and inspect how the /loadbalance/broker-time-average/ ZK statistics are produced and consumed. Compare the short-term and long-term values with the broker's reported CPU and traffic, then verify that threshold-based bundle shedding uses accurate statistics and does not incorrectly skip unloading.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.