nextflow-io / nextflow-io/nextflow
nf-azure: support Microsoft Fabric OneLake as object storage
Nobody has claimed this yet.
- Dominant language
- Groovy
- Stars
- 3.5k
- Forks
- 811
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 61
Description
New feature
nf-azure maps az://container/path to https://{account}.blob.core.windows.net. OneLake uses a different endpoint (https://onelake.dfs.fabric.microsoft.com, account onelake) and path layout (workspace / lakehouse / Files/). There is currently no way to use OneLake as workDir or for channel files.
Use case
Run pipelines with Azure Batch + Fusion + Entra managed identity, storing work and data in a Fabric lakehouse instead of a classic Blob container.
URI mapping: account = onelake, container = workspace, blob path = {item}.lakehouse/Files/{path}.
workDir = 'az://my-workspace/my-lakehouse.lakehouse/Files/nf-work'
Fabric constraints: Entra-only auth, lakehouse must already exist, I/O under Files/, GUIDs if names contain spaces.
Suggested implementation
Extend nf-azure (not a new plugin). Make the storage endpoint configurable (see #6580) and add azure.storage.kind = 'onelake' to translate az://workspace/item.lakehouse/Files/... to the OneLake URL, then pass that endpoint through to Fusion. Spike Blob SDK + managed identity against a real lakehouse first.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing nf-azure's existing storage path and issue #6580, then spike Blob SDK access with Entra managed identity against a real Fabric lakehouse. Verify the OneLake URI mapping, endpoint configuration, and Fusion handoff for workDir and channel files. Done means an existing lakehouse can be used under the stated Files/ and naming constraints.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, groovy
- Domain
- cloud
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100