nextflow-io / nextflow-io/nextflow
Support for more than 1 Azure Storage account
Nobody has claimed this yet.
- Dominant language
- Groovy
- Stars
- 3.5k
- Forks
- 811
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 61
Description
New feature
Azure currently only supports a single storage account. In a few instances, there is a need to work across multiple storage accounts. I've had to do this once, I have been contacted a few times by other users who wish to do this.
Usage scenario
Typically this is for one of a few reasons:
- Authentication reasons (e.g. access control via storage accounts).
- Only keeping a single copy of some files and using them in multiple places (e.g. reference data).
- Ingress/egress strategy (e.g. machine from a sequencer is then processed in the analysis storage account).
Suggest implementation
The actual how to do this isn't as hard as how to expose this to the pipeline developer and user. Somehow, we have to delineate between different storage accounts that have the same path and name, e.g. az://data/R1.fastq could exist twice. Furthermore, isolating this to a single storage account provides certain benefits (e.g. no cross-region data transfer, simpler access control). Perhaps extending authentication methods such as #3314 will make workarounds easier (e.g. use azcopy to transfer data between accounts within a process).
Consider this more of a discussion on practicality rather than a serious feature.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No files, tests, or entry points are named. Start by reviewing the existing Azure storage handling and the discussion around issue #3314; the work would be complete when multiple accounts can be distinguished for pipeline users while preserving account-specific access and transfer behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, groovy
- Domain
- cloud
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100