apache / apache/brpc

约千分之一的请求server耗时严重

Open
#2,094 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
17.6k
Forks
4.1k
Avg merge
2d 12h
Merged PRs (30d)
69

Description

**Describe the bug (描述bug)**
io-bound型服务,服务压力很低。查看工作线程数以及cpu使用程度都没有问题。bthread_worker_count:9,bthread_worker_usage维持在1.5以下。process_cpu_usage 0.5以下。排除锁的嫌疑。
使用rpcz发现耗时大的请求都卡在了最后Leave阶段,以下是其中一个,Leave core.HttpService.HotSearch阶段耗时350ms:
2023/01/18-14:58:29.320573 Received request(1378) from 127.0.0.1:53900 http log_id=0 trace=99c59dc851f7088b span=c51c916151762d93
14:58:29.320625 . 52 Processing the request in a new bthread
14:58:29.320672 . 47 Enter core.HttpService.HotSearch
14:58:29.320753 . 81 Requesting redis-server@xxx redis log_id=0 call_id=7142530642355 trace=99c59dc851f7088b span=[c51c916151762d94](http://xxx/rpcz?trace=99c59dc851f7088b&span=c51c916151762d94)
14:58:29.320770 . 17 Requested(49) [1]
14:58:29.322242 . 1472 Received response(14) of request[1]
14:58:29.322252 . 10 Processing the response in a new bthread
14:58:29.322256 . 4 Back to user's callsite
...
14:58:29.468671 . 17 Requested(467) [1]
14:58:29.469738 . 1067 Received response(5) of request[1]
14:58:29.469747 . 9 Processing the response in a new bthread
14:58:29.469751 . 4 Back to user's callsite
14:58:29.820964 .351213 Leave core.HttpService.HotSearch
14:58:29.821091 . 127 Responded(5280)

**To Reproduce (复现方法)**

**Expected behavior (期望行为)**

**Versions (各种版本)**
OS: CentOS Linux release 7.9.2009
Compiler: gcc8
brpc: 0.9.7
protobuf: 3.19

**Additional context/screenshots (更多上下文/截图)**

Contributor guide

Open the contributing guide

Research direction

Start by examining the rpcz trace for the long Leave core.HttpService.HotSearch interval and the surrounding request/response timing. Reproduce the rare delayed request if possible using the reported brpc 0.9.7, CentOS 7.9, GCC 8, and protobuf 3.19 versions; done means identifying why the Leave phase is delayed and documenting or fixing the cause.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
api, backend, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.