Use existing cluster resource managers for deployment of distributed workers

In GitLab by @pdamme on Oct 8, 2021, 20:18

The initial script for deploying distributed workers (#99) requires the manual specification of the worker nodes in terms of IP addresses/ports. It would be helpful to integrate the worker deployment with existing cluster resource managers, such as YARN.

Start by reviewing the initial distributed-worker deployment script referenced in issue #99 and the project’s current deployment entry points. Compare its manual IP/port configuration with the capabilities and requirements of YARN or another cluster resource manager; done means workers can be deployed through an existing manager without manually specifying worker nodes.

Written by the indexing model from the issue text.