[Performance] phi-moe is 3.84x slower than its FP16 reference

This was generated by AI during triage.

Summary

phi-moe completes aligned FP16 generation and matches the exact-text contract, but TRTMC latency is 3.84x the reference.

Observed behavior

• Precision: TRTMC FP16, reference FP16
• Output contract: exact text (matched)
• Request: batch 1, 10 new tokens
• Measurement: 3 warmups followed by 10 iterations
• Candidate p50: 921.89 ms
• Reference p50: 240.37 ms
• Regression: +283.5% (allowed margin: 5%)

Expected behavior

Mixture-of-experts generation should not incur a multi-fold runtime penalty under an otherwise equivalent workload.

Acceptance criteria

• Profile expert routing, expert kernels, prefill, and decode separately.
• Check for per-token host synchronization or expert fallback paths.
• Preserve exact-text output and the existing performance gate.

Reproduction platform

• Hardware: NVIDIA Thor X
• Architecture: aarch64
• TensorRT: 11.0
• GPU carveout: 48 GiB

Start by reproducing the batch-1 FP16 generation workload on NVIDIA Thor, then profile expert routing, expert kernels, prefill, and decode separately. Check for per-token host synchronization or expert fallback paths, while preserving exact-text output and the existing performance gate; done means the regression is reduced to the allowed 5% margin.