apache / apache/brpc

接口压测时 curl -i ip:port/rpcz 耗时 30s,gstack 查看 brpc 调用栈异常

Open
#2,189 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
17.6k
Forks
4.1k
Avg merge
2d 12h
Merged PRs (30d)
69

Description

**Describe the bug (描述bug)**
使用 jmeter 压测 brpc server 的某个接口(200并发),持续10分钟,过程中使用中间使用 curl -i ip:port/rpcz 查看接口执行情况,curl 约 30s 返回,接口普遍延时很高(正常时是100ms级,此时是10s级),此时通过 gstack 查看 brpc server 堆栈如下:
......
......
#16 0x00007f1d026d6803 in brpc::policy::ProcessHttpRequest(brpc::InputMessageBase*) () from ./libbrpc.so
#17 0x00007f1d0269a4da in brpc::ProcessInputMessage(void*) () from ./libbrpc.so
#18 0x00007f1d0269b283 in brpc::InputMessenger::OnNewMessages(brpc::Socket*) () from ./libbrpc.so
#19 0x00007f1d0274f57d in brpc::Socket::ProcessEvent(void*) () from ./libbrpc.so
#20 0x00007f1d025f381a in bthread::TaskGroup::task_runner(long) () from ./libbrpc.so
#21 0x00007f1d025dcf11 in bthread_make_fcontext () from ./libbrpc.so
#22 0x2be01d9a22db1800 in ?? ()
#23 0x2be01d9a4605fa49 in ?? ()
#24 0x025a3f014605fa49 in ?? ()
#25 0xe88ac60800000000 in ?? ()
#26 0x10019a8ef885b4cf in ?? ()
......
......
#264433 0x016300002c2d0000 in ?? ()
#264434 0x87192214f6000000 in ?? ()
#264435 0x0180aa94080180aa in ?? ()
#264436 0x00000000000063ce in ?? ()
#264437 0x0000000000000000 in ?? ()
Thread 57 (Thread 0x7f1cd79fe640 (LWP 65)):

**To Reproduce (复现方法)**
使用 jmeter 压测 brpc server 的某个接口(200并发)

**Expected behavior (期望行为)**
请求正常响应

**Versions (各种版本)**
OS: centos 7.6 编译,ubuntu22.04
Compiler: gcc 4.8.5
brpc: 1.1.0
protobuf:3.19.1

**Additional context/screenshots (更多上下文/截图)**

Contributor guide

Open the contributing guide

Research direction

Reproduce the issue with a brpc server under 200-concurrent JMeter load, then query ip:port/rpcz with curl -i while collecting gstack output. Start at ProcessHttpRequest, ProcessInputMessage, InputMessenger::OnNewMessages, Socket::ProcessEvent, and TaskGroup::task_runner. Done means identifying why rpcz takes about 30 seconds and requests reach second-level latency instead of responding normally.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
backend-api-design, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.