aliyun / aliyun/SimAI

CC_MODE 3仿真异常终止:assert failed

Open
#99 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.2k
Forks
184
Avg merge
13h 4m
Merged PRs (30d)
1

Description

各位老师好,请教个仿真异常终止的问题:
修改: astra-sim-alibabacloud/inputs/config/SimAI.conf中CC_MODE 3
启动:AS_SEND_LAT=3 AS_NVLS_ENABLE=1 ./bin/SimAI_simulator -t 96 -w ./aicb/results/workload/None-gpt_7B-world_size1024-tp4-pp1-ep1-gbs512-mbs1-seq4096-MOE-False-GEMM-False-flash_attn-True.txt -n ./HPN_7_0_1024_gpus_8_in_one_server_with_single_plane_100Gbps_A100 -c astra-sim-alibabacloud/inputs/config/SimAI.conf

输出:

info: no input grad collective for layer: grad_gather
Layer issue wg all gather at tick: 1027608530
chunk size is: 3261071360 , size is: 3261071360 , layer_num is: 0 , node: 0
info: all-gather weight grad collective issued for layer: grad_gather, involved dimensions:  0, 1, 1, 1, 1, 1, 1, 1, 1, 1,
pass: 0 finished at time: 1027608530
assert failed. cond="assert failed. cond="m_ptr", msg="Attempted to dereference zero pointer", m_ptrassert failed. cond="m_ptr", msg="Attempted to dereference zero pointerassert failed. cond="", msg="Attempted to dereference zero pointer", assert failed. cond="+1.027613444sm_ptr", msg="Attempted to dereference zero pointer", +1.027613444s 1216 file=/data/SimAI/astra-sim-alibabacloud/extern/network_backend/ns3-interface/simulation/build/include/ns3/ptr.h, line=649
terminate called without an active exception
assert failed. cond="m_ptr", msg="Attempted to dereference zero pointer+1.027613444 1194m_ptr", assert failed. cond="Aborted (core dumped)

万分感谢!

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the command with astra-sim-alibabacloud/inputs/config/SimAI.conf set to CC_MODE 3 and inspect the network backend at extern/network_backend/ns3-interface/simulation/build/include/ns3/ptr.h:649. Trace the failed pointer during the grad_gather/all-gather weight-grad path; done means the workload completes without the zero-pointer assertion.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.