Is the simulation deterministic?

Hello,

after seeing the issue #121 I've tried the following simulation twice with a single thread but the exposed communication time in the ncclFlowModel_EndToEnd.csv file changes a bit.

Topology: python3 ./astra-sim-alibabacloud/inputs/topo/gen_Topo_Template.py -topo DCN+ -g 32 -asn 2 -psn 2 -npa 16 -bw 400Gbps -l 1us

Workload: python3 -m workload_generator.AIOB_simAI_workload_generator --model_name GPT-13B --frame=Megatron --world_size=32 --tensor_model_parallel_size=2 --pipeline_model_parallel=1 --global_batch=32 --micro_batch=1 --num_layers=10 --seq_length=2048 --hidden_size=5120 --epoch_num=1 --use-distributed-optimizer --num_attention_heads=10 --aiob_enable --use_flash_attn --swiglu --comp_filepath workload/aiob_inputs/Example.txt

Simulation: AS_SEND_LAT=3 AS_NVLS_ENABLE=0 ./bin/SimAI_simulator -t 1 -w ./workloads/workload.txt -n ./topologies/topology -c astra-sim-alibabacloud/inputs/config/SimAI.conf

If I look at the ncclFlowModel_EndToEnd.csv file the exposed communication time of the two simulations is 211.5ms and 207.8ms.

To address this issue I've set the seed of the C++ standard library and the seed used by ns3 adding the following 2 lines of code in the main() of /SimAI/astra-sim-alibabacloud/astra-sim/network_frontend/ns3/AstraSimNetwork.cc, but the communication times still differ between multiple simulations.

int main(int argc, char *argv[]) {
  srand(12345);
  RngSeedManager::SetSeed(12345);
...

In other simulations the communication time changed even more, why this happens? Is the simulation deterministic?

Thanks in advance.

Reproduce the topology, workload, and simulation commands from the issue, then compare repeated ncclFlowModel_EndToEnd.csv results. Start at astra-sim-alibabacloud/astra-sim/network_frontend/ns3/AstraSimNetwork.cc and the seeding lines mentioned there; done means explaining whether runs are deterministic and identifying the source of any remaining variation.