nextflow-io / nextflow-io/nextflow

Azure Batch instance types automatic update

Open
#2,994 2 comments 0 reactions 1 assignee View on GitHub

@abhi18av is already working on this.

Since Oct 24, 2022.

executor/azure-batch
Dominant language
Groovy
Stars
3.5k
Forks
811
Avg merge
2d 11h
Merged PRs (30d)
61

Description

The Azure Batch support implemented by nextflow is taking the machine-type and location information from set of files embedded in the application itself.

https://github.com/nextflow-io/nextflow/blob/master/plugins/nf-azure/src/resources/nextflow/cloud/azure#L63-L63

However this is suboptimala because this information changes overtime and it's quite a hassle to be maintained manually

The goal of this issue is to implement an automatic update system that fetches this information from an Azure endpoint, and cache in the launching environment local disk.

The flow should be the following:

  1. check if the $NXF_HOME/azure/cache/instance-info.json exists
  2. if the file exists and it's not expired (older than 7 days) use it as a replacement of the current instance type and locations DB
  3. if the file does not exist or it's expired download the instance types endpoint, the continue at point 2
  4. if for any reason it cannot be downloaded fallback on the current database
Caveat
  • this feature should work with the existing Batch credentials i.e. the user should not be required for new keys to access those info

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.