google-deepmind / google-deepmind/xmanager
Multi-Worker Distributed Training
- Dominant language
- Python
- Stars
- 921
- Forks
- 69
- Avg merge
- 1d 1h
- Merged PRs (30d)
- 1
Description
I was wondering what exactly is the appropriate way to launch multi-worker distributed training jobs with xmanager. Based on my current understanding, it seems that a `Job` must be created for each worker pool and all these`Job`s must be combined into a single `JobGroup`, which is then added to the experiment. There also seems to be an option to add `Constraint`s for the `JobGroup`, however, I cannot seem to find what specific form these constraints may be able to take on besides the provided example of `xm_impl.SameMachine()`. Furthermore, my current attempt at launching a multi-worker distributing training job raises the following error when creating the distributed strategy with `strategy = tf.distribute.MultiWorkerMirroredStrategy()`: `RuntimeError: Collective ops must be configured at program startup`. Both the `CLUSTER_SPEC `and `TF_CONFIG` environment variables seem to be set correctly and the distributed strategy is created at the very beginning of the main function, so I was curious if this error might be due to the lack of setting appropriate `Constraint`s for the `JobGroup`.
Contributor guide
Assessment
This issue has not been assessed yet.