[ENHANCEMENT] Increasing throughput with vpp > 1

**Is your feature request related to a problem? Please describe.**
When we turn vpp > 1, I see a throughput drop as compared to not using vpp. The primary reason for this is the large "all_gather" latency incurred at the start of each vpp stage (maroon bars in image-1). Also, we see that the 1st “all_gather” operation for each vpp stage takes a long time, with subsequent “all_gather” happens instantly (2nd picture). So, for each model chunk corresponding to the vpp stages, the first time a batch flows through it, it takes time and all subsequent passes through the model chunk for all microbatches happens instantly.

The reason I believe that the 1st “all_gather” takes a long time, is because the first vpp stage is responsible for setting up the communication buffers and memory allocations, which might be taking that longer time. All subsequent batches reuse this established stream.

**Describe the solution you'd like**
I haven’t depp dived much into this. But, I would believe that by default Megatron/TE dont reuse the ‘all_gather’ buffers across vpp stages (hence the above phenemenon), such that each virtual stage is treated independently in terms of buffer lifecycle and microbatch processing. Instead, can we ensure that we pay this cost once per vpp stage, such that we dont pay this penalty for the next vpp model chunk on the same physical device? Basically, can we amortize the cost of the first expensive all_gather across all VPP stages.

**Describe alternatives you've considered**
We see the throughput drop if the number of microbatches aren't enough to cover for the time it takes for the all_Gather operations to take place across all the vpp stages, and this phenomenon only increases with vpp increasing (which is counter-intuitive).