apache / apache/brpc

开启Rpcz之后服务FATAL

Open
#630 4 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
C++
Stars
17.6k
Forks
4.1k
Avg merge
2d 12h
Merged PRs (30d)
69

Description

**Describe the bug (描述bug)**
FATAL

**To Reproduce (复现方法)**
1. 开启rpcz:
2. 在服务端发起异步请求并且不等待响应的到来(不Join)

复现的代码如下:

```
class EchoDone : public google::protobuf::Closure {
public:
void Run() {
std::unique_ptr self_guard(this);
}

brpc::Controller cntl;
example::EchoRequest req;
example::EchoResponse rsp;
};

namespace example {

class EchoServiceImpl : public EchoService {
public:
EchoServiceImpl() {
// 这里都使用example/echo_c++里的默认值,即single、baidu_std
// 下游是一个回射服务,在响应之前会bthread_usleep 100ms。
brpc::ChannelOptions options;
options.protocol = FLAGS_protocol;
options.connection_type = FLAGS_connection_type;
options.timeout_ms = FLAGS_timeout_ms;
options.max_retry = FLAGS_max_retry;
CHECK(client_channel_.Init(FLAGS_server.c_str(), FLAGS_load_balancer.c_str(), &options) == 0);
};

virtual ~EchoServiceImpl() {};
virtual void Echo(google::protobuf::RpcController* cntl_base,
const EchoRequest* request,
EchoResponse* response,
google::protobuf::Closure* done) {
brpc::ClosureGuard done_guard(done);

response->set_message(request->message());

EchoDone* async_done = new EchoDone;
example::EchoService_Stub stub(&client_channel_);
example::EchoRequest& req = async_done->req;
example::EchoResponse& rsp = async_done->rsp;
req.set_message("hello");
brpc::Controller& c = async_done->cntl;
// 在这里不Join该异步RPC,假如给下游服务设置一个延迟(如100ms),
// 那么async_done.cntl里的span_将很容易变成空悬指针,并触发fatal
stub.Echo(&c, &req, &rsp, async_done);
}

private:
brpc::Channel client_channel_;
};

```

原因分析是异步请求返回时,ServerSpan 已经被submit,controller里的 _span 变成了空悬指针。
此时该异步请求返回时调用的OnRPCEnd里再次submit,访问空悬指针,服务出现FATAL

**Expected behavior (期望行为)**
自动的不trace这些没有join的异步rpc,
或者
等到这些异步rpc返回之后再提交server span
目前我们在业务代码里,在发起不join的异步rpc之前,手动屏蔽了tls_bls::parent_span。等这些异步rpc发送之后再恢复parent_span的值,以此来绕过这个fatal

**Versions (各种版本)**
OS:
Compiler:
brpc: master
protobuf:

**Additional context/screenshots (更多上下文/截图)**
fatal时的调用栈如下,线上的情况更多是挂在 Collector::grab_thread() 这个函数

![image](https://user-images.githubusercontent.com/8801314/51815868-c7c31800-22fe-11e9-93fd-0504a9596ebf.png)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.