How to make it work with torch DDP

Debugging code for parallel training is very painful, and it would be very appealing to be able to do it in a notebook.

The most relevant thing I found is https://github.com/philtrade/Ddip and https://github.com/Bluefog-Lib/bluefog, but it seems that they are no longer being maintained.

Start by reviewing the issue's goal of making parallel training debuggable in a notebook, then compare the referenced Ddip and Bluefog projects and their maintenance status. Define the required torch DDP integration and a concrete notebook-based completion criterion before implementation.

由索引模型根據 Issue 內容生成。