约千分之一的请求server耗时严重
- Dominant language
- C++
- Stars
- 17.6k
- Forks
- 4.1k
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 69
Description
**Describe the bug (描述bug)**
io-bound型服务,服务压力很低。查看工作线程数以及cpu使用程度都没有问题。bthread_worker_count:9,bthread_worker_usage维持在1.5以下。process_cpu_usage 0.5以下。排除锁的嫌疑。
使用rpcz发现耗时大的请求都卡在了最后Leave阶段,以下是其中一个,Leave core.HttpService.HotSearch阶段耗时350ms:
2023/01/18-14:58:29.320573 Received request(1378) from 127.0.0.1:53900 http log_id=0 trace=99c59dc851f7088b span=c51c916151762d93
14:58:29.320625 . 52 Processing the request in a new bthread
14:58:29.320672 . 47 Enter core.HttpService.HotSearch
14:58:29.320753 . 81 Requesting redis-server@xxx redis log_id=0 call_id=7142530642355 trace=99c59dc851f7088b span=[c51c916151762d94](http://xxx/rpcz?trace=99c59dc851f7088b&span=c51c916151762d94)
14:58:29.320770 . 17 Requested(49) [1]
14:58:29.322242 . 1472 Received response(14) of request[1]
14:58:29.322252 . 10 Processing the response in a new bthread
14:58:29.322256 . 4 Back to user's callsite
...
14:58:29.468671 . 17 Requested(467) [1]
14:58:29.469738 . 1067 Received response(5) of request[1]
14:58:29.469747 . 9 Processing the response in a new bthread
14:58:29.469751 . 4 Back to user's callsite
14:58:29.820964 .351213 Leave core.HttpService.HotSearch
14:58:29.821091 . 127 Responded(5280)
**To Reproduce (复现方法)**
**Expected behavior (期望行为)**
**Versions (各种版本)**
OS: CentOS Linux release 7.9.2009
Compiler: gcc8
brpc: 0.9.7
protobuf: 3.19
**Additional context/screenshots (更多上下文/截图)**
Contributor guide
Research direction
Start by examining the rpcz trace for the long Leave core.HttpService.HotSearch interval and the surrounding request/response timing. Reproduce the rare delayed request if possible using the reported brpc 0.9.7, CentOS 7.9, GCC 8, and protobuf 3.19 versions; done means identifying why the Leave phase is delayed and documenting or fixing the cause.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- api, backend, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100