Azure / Azure/Batch

Support multi-instance tasks on concurrency enabled pools

Open
#109 8 comments 0 reactions 0 assignees View on GitHub
feature request
Dominant language
No language data
Stars
27
Forks
29
PR merge metrics
No merged PRs in 30d

Description

It's possible that I am not configuring my tasks correctly, because this seems like a feature that would already exist. But, I am having trouble getting RequiredSlots and multi-instance tasks to work.

When I create a batch job it contains thousands of tasks. I would say that 75% of the tasks created are small, and a single node could run 4 tasks concurrently. The remaining 25% are tasks that require many cores with MPI.

I had MPI working, but now need to implement the RequiredSlots feature in order to optimize our node usage.

I've refactored the small tasks to use RequiredSlots, and that seems to be working now. However, the MPI task that requires multiple nodes no longer executes.

The documentation states that the [requiredSlots property be set to 1](https://docs.microsoft.com/en-us/rest/api/batchservice/task/add) if it's a multi-instance task. I have tried setting this to 1, with the multi-instance property requiring 2 nodes, but the task never executes. I've also tried setting requiredSlots to the pool's TaskSlotsPerNode property (just to check), but of course received a validation exception stating the property value was invalid.

Are there any known issues with running a workflow that consists of tasks that consume just 1 slot (with each node having 4 slots), and other tasks that require multiple nodes with MPI?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.