AS_SEND_LAT has a bit impact on network busbw achieved

Hi SimAI team,

I find that the value of AS_SEND_LAT can impact a lot on the network busbw achieved when running this HPN 7.0 architecture VS DCN+ architecture example that mainly test the datacenter network communication.

For example, with AS_SEND_LAT value of 2 (microsecond), 512MB allreduce among 256 GPUs achieves 45.7 GB/s. But with default AS_SEND_LAT value of 6, the performance becomes 2.5 GB/s. So I wonder why the packet sending latency impacts so much on the achieved network bandwidth? What should be the right value for mimicking real-world collectives?

Another side question is: I saw the current gen_HPN_7.0_topo_mulgpus_one_link.py only generates two-layer datacenter networking topology (correct me if I am wrong). Is there any script I can use to test three-layer topology as described in the HPN sigcomm paper?

Best,
Yang

Start with the HPN 7.0 versus DCN+ example in docs/Tutorial.md and inspect how AS_SEND_LAT is used in the simulator. Read gen_HPN_7.0_topo_mulgpus_one_link.py to understand the generated topology, then compare the reported 2- and 6-microsecond results. Done means documenting the latency behavior and identifying whether a three-layer topology script exists or is needed.