Gang Scheduling

**Is this a BUG REPORT or FEATURE REQUEST?**: FEATURE REQUEST

Hi,

Does Argo plan to support Gang Scheduling? Let me detail what I mean by this.

Assume that someone would like to run a TensorFlow distributed training algorithm. This training algorithm needs 5 containers to be up and running before it can start.

Now imagine that various users would start 6 different instances of this workflow in parallel.

It is now possible that each workflow would start 3 containers, but none of the workflows would be able to start all 5 containers because the Kubernetes cluster would not have enough resources.

As a result, all the workflows are stuck.

Does Argo plan to provide a feature so that a workflow can start 5 containers if and only if it is able to get all five of them? Is there a way to circumvent this issue using Argo today? Maybe the workflow could release resources if it is not able to start all 5 containers within, say, 20 seconds. It would then retry later on using an exponential backoff strategy.

No files, tests, or entry points are named. Start by examining Argo Workflows' Kubernetes scheduling and workflow execution areas, then determine whether gang scheduling is supported or requires a new design. Done should be a clearly defined implementation scope or a documented, validated workaround.

Written by the indexing model from the issue text.