apache / apache/brpc

[RDMA] 启用RDMA后client侧解析响应失败

Open
#2,814 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
17.6k
Forks
4.1k
Avg merge
2d 12h
Merged PRs (30d)
69

Description

**Describe the bug (描述bug)**

应用场景可以简化为client发起rpc请求,server收到请求后取回数据,并通过response attachment返回,追踪日志看:
1. fuse->rpc->worker 完成
2. worker收到响应->take data-> append to attachment完成
3. fuse侧hang住,直到报错

```
E1031 07:32:53.252275 44 input_messenger.cpp:123] Fail to parse response from 33.51.173.89:19893 by baidu_std at client-side
W1031 07:32:53.252285 44 input_messenger.cpp:249] Close Socket{id=7 fd=1053 addr=33.51.173.89:19893:33384} (0x7f1d1765c780): absolutely wrong message
I1031 07:32:53.353038 43 socket.cpp:2606] Checking Socket{id=7 addr=33.51.173.89:19893} (0x7f1d1765c780)
I1031 07:32:53.353356 44 socket.cpp:2666] Revived Socket{id=7 addr=33.51.173.89:19893} (0x7f1d1765c780) (Connectable)
```

参考https://github.com/apache/brpc/issues/2265 对brpc收发方向的消息都打了日志,可见主要是正常的消息的和server所在进程,send方向的空数据:
![image](https://github.com/user-attachments/assets/25e143c0-ba72-4d9c-a5e2-077829449488)

worker响应的代码很简单,blockletBuffer打印出之后确实是有数据的,没有进一步的排查头绪了。

```cpp
_cntl->response_attachment().append(blockletBuffer->data(), blockletBuffer->size());
```

**To Reproduce (复现方法)**

**Expected behavior (期望行为)**

**Versions (各种版本)**
OS:
Compiler:
brpc: 1.11.0
protobuf:

**Additional context/screenshots (更多上下文/截图)**

Contributor guide

Open the contributing guide

Research direction

Start with input_messenger.cpp and socket.cpp, then compare the send/receive logging discussed in issue #2265. Trace the response_attachment().append call and the client-side response parsing around the reported RDMA failure. Done means reproducing the failure and identifying why the client treats the response as an invalid message.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
backend-api-design, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.