nextflow-io / nextflow-io/nextflow

Support reservation when provisioning VM/ accelerators on Google Batch

Open
#4,262 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

executor/google-batch good first issue stale
Dominant language
Groovy
Stars
3.5k
Forks
811
Avg merge
2d 11h
Merged PRs (30d)
61

Description

New feature

Google Cloud supports provisioning Compute Engine instances using reservations (e.g. dedicated compute resources CPUs/GPUs/Memory etc).
https://cloud.google.com/compute/docs/instances/reservations-consume

This is a configuration available via Google LifeSciences API and Google Batch.

reservation stringIf specified, VMs will consume only the specified reservation. If not specified (default), VMs will consume any applicable reservation.
reservation stringIf specified, the VM will only be allocated inside the matching reservation. It will fail if the VM parameters don't match the reservation.

Usage scenario

For example a user may configure a reservation for GPU resources to ensure dedicated resources are available to be provisioned (instead of waiting for GPUs to be available on demand).

Reservations can be made and applied automatically but it can be useful to be able to explicitly specify a reservation in case it's only used by a certain user or for a certain project.

Then the nextflow user would want to be able to configure a particular Nextflow process to use that dedicated reservation when provisioning a task that uses a GPU.

Suggest implementation

Allow for a Nextflow process to specify reservation configuration along with machineType/accelerators etc.

process foo {
    accelerator 4, type: 'nvidia-tesla-k80'
    reservation: 'userx-gpu-reservation'

    script:
    """
    your_gpu_enabled --command --line
    """
}

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the existing Google Cloud configuration for process machineType and accelerators, then compare the Google Batch InstancePolicy and Life Sciences VirtualMachine reservation fields linked in the issue. Define how a process-level reservation value should be represented and passed to the selected provisioning API. Done means a configured reservation is honored for the relevant VM or accelerator task, with coverage for the supported Google Cloud path.

Written by the indexing model from the issue text.

Assessment

Tech stack
google-cloud, groovy
Domain
cloud
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.