apache / apache/brpc

[E1008]Reached timeout=10000ms

Open
#2,686 2 comments 1 reaction 0 assignees View on GitHub
Dominant language
C++
Stars
17.6k
Forks
4.1k
Avg merge
2d 12h
Merged PRs (30d)
69

Description

**Describe the bug (描述bug)**
brpc客户端在连接空闲了一段时间后,大约1.5h,再次请求发现会会报这样的错误 `[E1008]Reached timeout=10000ms`,大约再过了15分钟后,问题不再出现。

**To Reproduce (复现方法)**
brpc客户端在连接空闲了一段时间后,大约1.5h,再次请求发现会会报这样的错误 `[E1008]Reached timeout=10000ms`

**Expected behavior (期望行为)**
正常请求,不报E1008错误

**Versions (各种版本)**
OS: centos7.9
Compiler: gcc 9.3.1
brpc: -
protobuf: -

**Additional context/screenshots (更多上下文/截图)**
``` cpp
Client::Client() {
auto config = ConfigManager::GetConfigManager();

brpc::ChannelOptions options;

options.protocol = config->GetConfigParam(
"risk_control", "protocol", "baidu_std");
options.connection_type = config->GetConfigParam(
"risk_control", "connection_type", "");
options.timeout_ms =
config->GetConfigParam("risk_control", "timeout_ms", "100");
options.max_retry =
config->GetConfigParam("risk_control", "max_retry", "3");

const std::string server = config->GetConfigParam(
"risk_control", "server", "0.0.0.0:8000");
const std::string load_balancer =
config->GetConfigParam("risk_control", "load_balancer", "");

if (channel_.Init(server.c_str(), load_balancer.c_str(), &options) != 0) {
SLOG_ERROR("", "RiskControl", "Client Init failed", 0, 0);
}
stub_.reset(new rc::AsyncRcService_Stub(&channel_));
}
```
初步分析了下,应该是tcp连接假死导致应用层没有感知到,触发了TCP的超时重传。
![image](https://github.com/apache/brpc/assets/34154239/ea58c529-4b60-4aaf-97f7-02fd78964b13)

通过`lsof`命令查看客户端和服务端的TCP连接都是`established`的状态,所以比较疑惑:
- 什么场景下会出现tcp假死呢?
- tcp假死如何快速解脱呢?

Contributor guide

Open the contributing guide

Research direction

Start with the brpc client setup shown in the issue, especially ChannelOptions, timeout_ms, max_retry, and the connection_type setting. Investigate how an idle TCP connection can remain established while requests reach the 10000ms E1008 timeout; done means reproducing the idle-connection failure and documenting the cause and a reliable way to recover or avoid it.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.