aliyun / aliyun/SimAI

pfc触发段错误

Open
#286 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.2k
Forks
184
Avg merge
13h 4m
Merged PRs (30d)
1

Description

Image Image

我将配置文件中的PACKET_PAYLOAD_SIZE 9000 改为1000,将buffer_size从32改为9,其余不变相同的一个workload,执行后者报错如图,AI告诉我是pfc resume的时候输入的队列id不对导致的,存在这个问题吗。

PFC 触发链路是:

SwitchNode::SendToDev()
-> CheckAndSendPfc(inDev, qIndex)
-> device->SendPfc(qIndex, 0) // pause

恢复链路是:

SwitchNode::SwitchNotifyDequeue()
-> CheckAndSendResume(inDev, qIndex)
-> device->SendPfc(qIndex, 1) // resume

GPU/NIC 收到 resume 后:

QbbNetDevice::Receive()
-> Resume(qIndex)

问题就在 Resume():

Ptr lastQp = m_rdmaEQ->GetQp(qIndex);

这里的 qIndex 是 PFC 队列号/优先级号,比如 3,不是 RDMA QP 数组下标。
你这次 gdb 里已经看到:

qIndex = 3
node id = 10
node type = 0
RDMA QP count = 2

也就是只有 2 个 QP,却访问 GetQp(3),直接越界段错误。

为什么 BUFFER_SIZE=9 更容易触发?

PFC 阈值这样算:

(buffer_size - total_hdrm - total_rsrv - shared_used_bytes) >> pfc_a_shift[port]

BUFFER_SIZE 从 32MB 改成 9MB,buffer_size 变小,PFC threshold 明显变低,更容易 pause,也更频繁 resume。于是更容易走到这个错误的 Resume(qIndex)。

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at QbbNetDevice::Receive() and Resume(), then trace the qIndex passed from SwitchNode::SendToDev() and SwitchNotifyDequeue() through device->SendPfc(). Reproduce the reported workload with PACKET_PAYLOAD_SIZE 1000 and buffer_size 9, inspect the RDMA QP count at GetQp(qIndex), and confirm that resume handling completes without an out-of-range access.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
networking
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.