apache / apache/brpc

求助:卡死问题排查

Open
#2,552 6 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
17.6k
Forks
4.1k
Avg merge
2d 12h
Merged PRs (30d)
69

Description

**Describe the bug (描述bug)**
adx服务在启动的时候大概率会出现卡死。vars接口显示创建了大量bthread和channel_connections,并且不断增长,最后导致OOM。如果能成功启动,服务运行还是比较稳定,高峰时bthread最多1800个,channel_connections最多2500个。

brpc服务接收到请求后会广播给下游dsp。协议基本都是http。我们对每一个dsp请求都会启一个bthread,bthread里同步请求下游服务,设定的超时时间为500ms左右。处理服务请求的bthread会join每个dsp的bthread。

服务最高并发不到100。dsp120个左右。

gdb调试的时候,worker线程堆栈目前看到有两种:

1. worker都在发送dsp请求,暂停在dsp的CallMethod。看堆栈貌似把pthread卡住了
![image](https://github.com/apache/brpc/assets/598529/239c9e1b-407c-4315-b13d-8ebc7771c7b1)

2. worker都在建立ssl链接
![image](https://github.com/apache/brpc/assets/598529/d62949a9-b234-45d1-a53a-293e57146d2e)

有没有排查思路?

**To Reproduce (复现方法)**

**Expected behavior (期望行为)**

**Versions (各种版本)**
OS: ubuntu 22.04.3
Compiler: g++ 11.4
brpc: 1.6
protobuf: 3.20.1

**Additional context/screenshots (更多上下文/截图)**

Contributor guide

Open the contributing guide

Research direction

Start with the vars output and the two GDB worker-stack patterns described: blocked DSP CallMethod calls and SSL connection setup. Inspect how the service creates and joins bthreads for each downstream DSP request, and how channel_connections grow during startup. Done means identifying the cause of the unbounded growth or documenting a reproducible diagnosis and concrete mitigation.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
backend, distributed-systems, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.