SimAI_analytical hangs / becomes extremely slow for tp=2 configs

Hi, I ran into a strange performance issue with SimAI_analytical.

Environment

• SimAI commit: f5efb5a
• Ubuntu 24.04.4 LTS
• GCC 13.3.0
• Built with ./scripts/build.sh -c analytical

I'm running a 256-GPU Megatron training workload.

For example:

• dp=8, tp=4, pp=8 finishes in about 1 second
• dp=8, tp=2, pp=16 did not finish after 2–3 hours

Other configs with tp >= 4 also finish within a few seconds. So far I only see this behavior with tp=2.

The slow case does not have a larger workload either:

• tp=2: 781 workload entries, 776 ALLREDUCE ops
• tp=4: 1421 workload entries, 1416 ALLREDUCE ops
• tp=8: 3085 workload entries, 3080 ALLREDUCE ops

Memory usage stays around 5 MB during the slow run, while one CPU core stays busy.

I traced it to the event loop in:

astra-sim-alibabacloud/astra-sim/network_frontend/analytical/AnaSim.cc

queue<struct CallTask> call_list;
uint64_t tick = 0;

void AnaSim::Run() {
    while (!call_list.empty()) {
        CallTask calltask = call_list.front();

        while (true) {
            if (calltask.time != tick) {
                tick++;
            } else {
                break;
            }
        }

        call_list.pop();
        calltask.fun_ptr(calltask.fun_arg);
    }
}

A couple of things here look suspicious to me:

1. tick is incremented one by one until it reaches calltask.time. If the timestamp is large, this can result in a very large number of iterations.

2. call_list is a FIFO queue. If an event with calltask.time < tick is popped, tick++ can no longer reach that value.

I also noticed that Sys::generate_time() takes int cycles:

timespec_t generate_time(int cycles);

I'm not sure if this is related, but some delay values seem to come from 64-bit values.

Has anyone seen this behavior before? Is this expected in the analytical backend?

Thanks for taking a look!