GPU number inconsistency between topology file and workload
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.2k
- Forks
- 184
- Avg merge
- 13h 4m
- Merged PRs (30d)
- 1
Description
I followed the SimAI-Simulation Tutorial and it works:
# Create network topo
$ python3 ./astra-sim-alibabacloud/inputs/topo/gen_HPN_7.0_topo_mulgpus_one_link.py -g 128 -gt A100 -bw 100Gbps -nvbw 2400Gbps
# Running
$ AS_SEND_LAT=3 AS_NVLS_ENABLE=1 ./bin/SimAI_simulator -t 16 -w ./example/microAllReduce.txt -n ./HPN_7_0_128_gpus_8_in_one_server_with_single_plane_100Gbps_A100 -c astra-sim-alibabacloud/inputs/config/SimAI.conf
It seems that the total gpu number is 128 in the topology file. However, the gpu number in ./example/microAllReduce.txt is only 8. What is the reason of this inconsistency?
Besides, when I tried to generate a workload of 128 gpus:
./bin/SimAI_analytical \
-w /workspace/aicb/results/workload/-gpt_7B-world_size128-tp4-pp1-ep1-gbs16-mbs1-seq16-MOE-False-GEMM-False-flash_attn-True.txt \
-g 128 \
-g_p_s 8 \
-r test-analytical \
-busbw example/busbw.yaml \
-v
and run the Sim-AI-Simulation
python3 ./astra-sim-alibabacloud/inputs/topo/gen_HPN_7.0_topo_mulgpus_one_link.py \
-g 128 \
-gt A100 \
-gps 8 \
-bw 100Gbps \
-nvbw 2400Gbps
AS_NVLS_ENABLE=1 AS_SEND_LAT=3 ./bin/SimAI_simulator \
-t 32 \
-w /workspace/aicb/results/workload/-gpt_7B-world_size128-tp4-pp1-ep1-gbs16-mbs1-seq16-MOE-False-GEMM-False-flash_attn-True.txt \
-n /workspace/SimAI/HPN_7_0_128_gpus_8_in_one_server_with_single_plane_100Gbps_A100 \
-c astra-sim-alibabacloud/inputs/config/SimAI.conf
I could only get empty output csv files.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by comparing the GPU-count arguments and generated topology name in astra-sim-alibabacloud/inputs/topo/gen_HPN_7.0_topo_mulgpus_one_link.py with the workload files passed to SimAI_simulator. Then inspect astra-sim-alibabacloud/inputs/config/SimAI.conf and the simulator run output to determine why the 128-GPU workload produces empty CSV files. Done means the expected GPU-count relationship is documented or corrected and the reproduction produces meaningful output.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ai-infra-agents, hpc
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100