nextflow-io / nextflow-io/nextflow

Support for more than 1 Azure Storage account

Open
#4,445 15 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

storage/azure
Dominant language
Groovy
Stars
3.5k
Forks
811
Avg merge
2d 11h
Merged PRs (30d)
61

Description

New feature

Azure currently only supports a single storage account. In a few instances, there is a need to work across multiple storage accounts. I've had to do this once, I have been contacted a few times by other users who wish to do this.

Usage scenario

Typically this is for one of a few reasons:

  • Authentication reasons (e.g. access control via storage accounts).
  • Only keeping a single copy of some files and using them in multiple places (e.g. reference data).
  • Ingress/egress strategy (e.g. machine from a sequencer is then processed in the analysis storage account).

Suggest implementation

The actual how to do this isn't as hard as how to expose this to the pipeline developer and user. Somehow, we have to delineate between different storage accounts that have the same path and name, e.g. az://data/R1.fastq could exist twice. Furthermore, isolating this to a single storage account provides certain benefits (e.g. no cross-region data transfer, simpler access control). Perhaps extending authentication methods such as #3314 will make workarounds easier (e.g. use azcopy to transfer data between accounts within a process).

Consider this more of a discussion on practicality rather than a serious feature.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No files, tests, or entry points are named. Start by reviewing the existing Azure storage handling and the discussion around issue #3314; the work would be complete when multiple accounts can be distinguished for pipeline users while preserving account-specific access and transfer behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, groovy
Domain
cloud
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.