nextflow-io / nextflow-io/nextflow

Support accelerator directive for Azure Batch

Open
#3,789 13 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

executor/azure-batch stale
Dominant language
Groovy
Stars
3.5k
Forks
811
Avg merge
2d 11h
Merged PRs (30d)
61

Description

The accelerator directive is currently supported for AWS Batch, Google Batch, and Kubernetes. Azure Batch also has GPUs, so this directive should also work for Azure Batch.

Not sure yet how to implement it yet with the Azure Batch Java SDK. All I found so far is the GpuResource class. @abhi18av do you have any idea how we would support GPU requests here? I don't know if there are specific GPU-enabled instance types, or if we can just attach a GPU to any instance or what.

In any case, I think we can map the accelerator type to the GpuSku enum:

final accel = task.config.getAccelerator()
new GpuResource()
    .withCount( accel.getRequest() )
    .withType( GpuSku.fromString(accel.getType()) )

So an example request would look like:

accelerator 1, type: 'V100'

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the existing accelerator support for AWS Batch, Google Batch, and Kubernetes, then inspect how the Azure Batch Java SDK represents GPU resources and instance types. Determine whether GpuResource and GpuSku apply to Azure Batch, and define the supported mapping for an example such as accelerator 1, type: 'V100'. Done means Azure Batch accepts and provisions the requested accelerator consistently with the other backends.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, groovy
Domain
cloud
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.