Feature Request: Enhanced Cross-Datacenter Training UX
- Dominant language
- Python
- Stars
- 17.9k
- Forks
- 4.5k
- Avg merge
- 4d 6h
- Merged PRs (30d)
- 271
Description
## Summary
Request improved user experience for multi-datacenter LLM training via dedicated CLI options, documentation, and example configs. The HSDP foundation exists in Megatron Core v0.11.0, but users must manually configure complex DeviceMesh setups without guidance.
## Motivation
Multi-datacenter training is becoming essential for scaling. The [NVIDIA blog](https://developer.nvidia.com/blog/turbocharge-llm-training-across-long-haul-data-center-networks-with-nvidia-nemo-framework/) demonstrates 96% scaling efficiency training a 340B model across data centers ~1000 km apart. However:
- **High configuration complexity** - Users must manually configure HSDP with DeviceMesh, understand `dp_outer`/`dp_shard` relationships, and tune communication parameters for WAN environments
- **Missing documentation** - "N/S connection" mentioned in CHANGELOG v0.11.0 but never explained; no user guide or examples for cross-datacenter setups
- **No bandwidth/latency-aware tuning** - No automatic adjustment of chunk sizes or communication strategies based on network topology
- **Limited observability** - No metrics distinguishing inter-DC vs intra-DC communication patterns
## Current State
Megatron Core has HSDP (Hybrid Sharded Data Parallel) implementation in `megatron/core/distributed/fsdp/` and basic CLI but lacks user-friendly abstractions for common cross-datacenter deployment patterns.
## Ask
1. **New CLI arguments** for intuitive multi-datacenter configuration
2. **Documentation** - User guide at `docs/user-guide/features/multi_datacenter_training.md` explaining architecture, configuration, and tuning
3. **Example scripts** - `examples/cross_datacenter/` with working configurations for common setups (2-DC, multi-DC with HSDP)
4. **Monitoring** - wandb metrics for inter-DC vs intra-DC communication breakdown
## References
- [NVIDIA Blog: Turbocharge LLM Training Across Long-Haul Data Center Networks](https://developer.nvidia.com/blog/turbocharge-llm-training-across-long-haul-data-center-networks-with-nvidia-nemo-framework/)
- [Megatron-FSDP README](https://github.com/NVIDIA/Megatron-LM/tree/main/megatron/core/distributed/fsdp/src)
Contributor guide
Assessment
This issue has not been assessed yet.