[Feature]: Reduce model compilation time

🚀 The feature, motivation and pitch

Need to understand better how AD spends time during compilation.

1. Analyze where time is spent in each pass
2. Do we need all cleanup passes to run each time?
3. Are we spending a lot of time on i/o?

Below is an example of DS-R1 compilation time.
In this example the vast majority of time (45 mins - 50%) is spent on sharding_transform_executor

time python examples/auto_deploy/build_and_run_ad.py --model deepseek-ai/DeepSeek-R1 --args.world-size 8 --args.skip-loading-weights true --args.attn_backend triton

90 mins from start to failure (real 93m59.568s)

[2025-08-25 04:31:13] start
...fetch checkpoint etc.
[08/25/2025-04:31:58] [TRT-LLM AUTO-DEPLOY] [RANK 3] [I] stage=factory, transform=build_model, num_matches=1, is_clean=False, has_valid_shapes=False
[08/25/2025-04:41:20] [TRT-LLM AUTO-DEPLOY] [RANK 3] [I] stage=export, transform=export_to_gm, num_matches=1, is_clean=False, has_valid_shapes=False
[08/25/2025-04:45:30] [TRT-LLM AUTO-DEPLOY] [RANK 3] [I] stage=post_export, transform=cleanup_noop_slice, num_matches=734, is_clean=True, has_valid_shapes=False
[08/25/2025-04:48:17] [TRT-LLM AUTO-DEPLOY] [RANK 3] [I] stage=post_export, transform=cleanup_noop_add, num_matches=0, is_clean=True, has_valid_shapes=False
[08/25/2025-04:49:34] [TRT-LLM AUTO-DEPLOY] [RANK 3] [I] stage=post_export, transform=cleanup_input_constraints, num_matches=2, is_clean=True, has_valid_shapes=False
[08/25/2025-04:52:30] [TRT-LLM AUTO-DEPLOY] [RANK 3] [I] stage=pattern_matcher, transform=match_moe_pattern, num_matches=0, is_clean=True, has_valid_shapes=False
[08/25/2025-04:54:39] [TRT-LLM AUTO-DEPLOY] [RANK 3] [I] stage=pattern_matcher, transform=match_repeat_kv, num_matches=0, is_clean=True, has_valid_shapes=False
[08/25/2025-04:56:10] [TRT-LLM AUTO-DEPLOY] [RANK 3] [I] stage=pattern_matcher, transform=match_eager_attention, num_matches=61, is_clean=True, has_valid_shapes=False
[08/25/2025-04:58:17] [TRT-LLM AUTO-DEPLOY] [RANK 3] [I] stage=pattern_matcher, transform=match_grouped_attention, num_matches=61, is_clean=True, has_valid_shapes=False
[08/25/2025-05:00:41] [TRT-LLM AUTO-DEPLOY] [RANK 3] [I] stage=pattern_matcher, transform=match_attention_layout, num_matches=61, is_clean=True, has_valid_shapes=False
[08/25/2025-05:02:35] [TRT-LLM AUTO-DEPLOY] [RANK 3] [I] stage=pattern_matcher, transform=match_rope_pattern, num_matches=61, is_clean=True, has_valid_shapes=False
[08/25/2025-05:05:09] [TRT-LLM AUTO-DEPLOY] [RANK 3] [I] stage=pattern_matcher, transform=match_rope_layout, num_matches=61, is_clean=True, has_valid_shapes=False
[08/25/2025-05:07:31] [TRT-LLM AUTO-DEPLOY] [RANK 3] [I] stage=pattern_matcher, transform=eliminate_redundant_transposes, num_matches=122, is_clean=True, has_valid_shapes=False
[08/25/2025-05:09:45] [TRT-LLM AUTO-DEPLOY] [RANK 3] [I] stage=pattern_matcher, transform=optimize_rope, num_matches=0, is_clean=True, has_valid_shapes=False
[08/25/2025-05:09:45] [TRT-LLM AUTO-DEPLOY] [RANK 3] [I] stage=pattern_matcher, transform=quantize_from_config, skipped=True, is_clean=True, has_valid_shapes=False
[08/25/2025-05:09:45] [TRT-LLM AUTO-DEPLOY] [RANK 3] [I] stage=pattern_matcher, transform=quantize_from_graph, skipped=True, is_clean=True, has_valid_shapes=False
[08/25/2025-05:09:46] [TRT-LLM AUTO-DEPLOY] [RANK 3] [I] stage=pattern_matcher, transform=quantize_moe, skipped=True, is_clean=True, has_valid_shapes=False
[08/25/2025-05:12:08] [TRT-LLM AUTO-DEPLOY] [RANK 3] [I] stage=sharding, transform=detect_column_row_shard, num_matches=123, is_clean=True, has_valid_shapes=False
[08/25/2025-05:13:31] [TRT-LLM AUTO-DEPLOY] [RANK 3] [I] stage=sharding, transform=detect_ep_shard, num_matches=58, is_clean=True, has_valid_shapes=False
[08/25/2025-05:15:23] [TRT-LLM AUTO-DEPLOY] [RANK 3] [I] stage=sharding, transform=detect_dp_bmm_shard, num_matches=0, is_clean=True, has_valid_shapes=False
[08/25/2025-05:59:31] [TRT-LLM AUTO-DEPLOY] [RANK 3] [I] stage=sharding, transform=sharding_transform_executor, num_matches=605, is_clean=True, has_valid_shapes=True
[08/25/2025-05:59:31] [TRT-LLM AUTO-DEPLOY] [RANK 3] [I] Loading and initializing weights.
[08/25/2025-05:59:32] [TRT-LLM] [RANK 3] [E] Failed to initialize executor on rank 3: CUDA out of memory.

Alternatives

No response

Additional context

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and checked the documentation and examples for answers to frequently asked questions.

Start with examples/auto_deploy/build_and_run_ad.py and reproduce the logged DeepSeek-R1 compilation command. Trace the compilation passes, especially sharding_transform_executor, to measure pass and I/O time; the issue does not define a concrete optimization or completion criteria.