nextflow-io / nextflow-io/nextflow
Azure Batch instance types automatic update
Open
@abhi18av is already working on this.
Since Oct 24, 2022.
executor/azure-batch
- Dominant language
- Groovy
- Stars
- 3.5k
- Forks
- 811
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 61
Description
The Azure Batch support implemented by nextflow is taking the machine-type and location information from set of files embedded in the application itself.
However this is suboptimala because this information changes overtime and it's quite a hassle to be maintained manually
The goal of this issue is to implement an automatic update system that fetches this information from an Azure endpoint, and cache in the launching environment local disk.
The flow should be the following:
- check if the
$NXF_HOME/azure/cache/instance-info.jsonexists - if the file exists and it's not expired (older than 7 days) use it as a replacement of the current instance type and locations DB
- if the file does not exist or it's expired download the instance types endpoint, the continue at point 2
- if for any reason it cannot be downloaded fallback on the current database
Caveat
- this feature should work with the existing Batch credentials i.e. the user should not be required for new keys to access those info
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.