google-deepmind / google-deepmind/xmanager

Multi-Worker Distributed Training

Open
#16 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
921
Forks
69
Avg merge
1d 1h
Merged PRs (30d)
1

Description

I was wondering what exactly is the appropriate way to launch multi-worker distributed training jobs with xmanager. Based on my current understanding, it seems that a `Job` must be created for each worker pool and all these`Job`s must be combined into a single `JobGroup`, which is then added to the experiment. There also seems to be an option to add `Constraint`s for the `JobGroup`, however, I cannot seem to find what specific form these constraints may be able to take on besides the provided example of `xm_impl.SameMachine()`. Furthermore, my current attempt at launching a multi-worker distributing training job raises the following error when creating the distributed strategy with `strategy = tf.distribute.MultiWorkerMirroredStrategy()`: `RuntimeError: Collective ops must be configured at program startup`. Both the `CLUSTER_SPEC `and `TF_CONFIG` environment variables seem to be set correctly and the distributed strategy is created at the very beginning of the main function, so I was curious if this error might be due to the lack of setting appropriate `Constraint`s for the `JobGroup`.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.