nextflow-io / nextflow-io/nextflow

nf-azure: support Microsoft Fabric OneLake as object storage

Open
#7,495 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

storage/azure
Dominant language
Groovy
Stars
3.5k
Forks
811
Avg merge
2d 11h
Merged PRs (30d)
61

Description

New feature

nf-azure maps az://container/path to https://{account}.blob.core.windows.net. OneLake uses a different endpoint (https://onelake.dfs.fabric.microsoft.com, account onelake) and path layout (workspace / lakehouse / Files/). There is currently no way to use OneLake as workDir or for channel files.

Use case

Run pipelines with Azure Batch + Fusion + Entra managed identity, storing work and data in a Fabric lakehouse instead of a classic Blob container.

URI mapping: account = onelake, container = workspace, blob path = {item}.lakehouse/Files/{path}.

    workDir = 'az://my-workspace/my-lakehouse.lakehouse/Files/nf-work'

Fabric constraints: Entra-only auth, lakehouse must already exist, I/O under Files/, GUIDs if names contain spaces.

Suggested implementation

Extend nf-azure (not a new plugin). Make the storage endpoint configurable (see #6580) and add azure.storage.kind = 'onelake' to translate az://workspace/item.lakehouse/Files/... to the OneLake URL, then pass that endpoint through to Fusion. Spike Blob SDK + managed identity against a real lakehouse first.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing nf-azure's existing storage path and issue #6580, then spike Blob SDK access with Entra managed identity against a real Fabric lakehouse. Verify the OneLake URI mapping, endpoint configuration, and Fusion handoff for workDir and channel files. Done means an existing lakehouse can be used under the stated Files/ and naming constraints.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, groovy
Domain
cloud
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.