Feature Request: Configurable Communication Scheduling Policies for MoE Training

## Feature Request: Configurable Communication Scheduling Policies for MoE Training

### Motivation

Recent work and practical experience with MoE training suggest that communication scheduling policies can have a noticeable impact on end-to-end performance.

In Megatron, communication operations such as:

* EP All-to-All
* DP gradient All-Reduce / Reduce-Scatter
* TP collectives

are typically dispatched using fixed stream assignments and overlap strategies implemented inside the runtime.

While this works well in many cases, it makes it difficult to experimentally evaluate alternative scheduling choices without modifying internal implementations.

For example, when EP All-to-All and DP reductions contend for the same network resources, different policies may lead to different trade-offs between overlap and network interference.

### Proposed Feature

Expose a configurable communication scheduling policy through runtime arguments.

Example:

```bash
--comm-schedule-policy separate
```

Possible policies:

* `separate` (default)

* Use the current Megatron behavior.
* Dispatch collectives to their existing streams for maximum overlap.

* `serialized`

* Execute selected communication operations sequentially on the same stream.
* Useful for evaluating whether reducing network contention improves throughput.

Potential future extensions:

* `bucketized`

* Split DP reductions into smaller buckets to enable finer-grained overlap.

### Why This Could Be Useful

This would provide researchers and practitioners with an easy way to evaluate communication scheduling strategies on different hardware topologies without invasive runtime modifications.

Possible evaluation scenarios include:

* Mixtral/Qwen-MoE style models;
* varying EP/DP configurations;
* different interconnects (NVLink, InfiniBand, RoCE);
* DualPipe-like schedules.

### Backward Compatibility

The default behavior (`separate`) would preserve the current implementation and introduce no performance changes for existing users.

### Related Context

This request is motivated by recent discussions in the systems community around exposing communication scheduling decisions as configurable policies rather than hard-coded runtime behavior, enabling easier experimentation and benchmarking.

Would the maintainers be open to such an abstraction if implemented incrementally and with benchmarks demonstrating its usefulness?