Validate async tensor parallelism for the main SFT path

Goal

Make async tensor parallelism a supported and verified option for the standard LLM SFT recipe.

AutoModel already has the async-TP configuration and graph-shaping work from #1711 and #2987. This issue is limited to validating the main path and making only the minimal fixes exposed by that validation.

Required scope

Use one representative dense BF16 Llama SFT configuration through TrainFinetuneRecipeForNextTokenPrediction with:

• FSDP2 and TP > 1
• sequence_parallel: true
• per-layer torch.compile
• fixed-length, non-packed inputs
• PP1 and CP1

Compare regular TP and async TP with the same model, checkpoint, data, seed, and compile settings.

Acceptance criteria

• The SFT run completes with finite loss when async TP is enabled.
• Loss and the training update agree with regular TP within an appropriate BF16 tolerance.
• Confirm Inductor emits fused_all_gather_matmul and fused_matmul_reduce_scatter.
• Show a repeatable step-time or throughput gain over regular TP after compile warm-up.
• Add or update one focused test or benchmark recipe and document the required configuration.
• Apply only minimal fixes uncovered by this validation.

Optional, not release-blocking

• A second SFT mode such as full-weight SFT if LoRA is used for the primary validation
• Packed or dynamic sequence lengths
• Pipeline-parallel or context-parallel combinations
• FP8 or other quantized SFT

Out of scope

• Custom MoE tensor parallelism
• QLoRA or BitsAndBytes
• MegatronFSDP or DDP async-TP support

References

• #1711
• #2987
• TorchTitan async-TP compile setup