microsoft / microsoft/finops-toolkit

Fabric Eventhouse ingestion fails with private storage account firewall

Open
#2,061 2 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Resolution: External Tool: FinOps hubs
Dominant language
PowerShell
Stars
603
Forks
248
Avg merge
7d 11h
Merged PRs (30d)
11

Description

🐛 Problem

When deploying FinOps Hub v13 with Fabric Eventhouse and a storage account with firewall enabled (public access disabled), ingestion fails because Fabric Eventhouse cannot read parquet files from the storage account.

The root cause is that ADF sends a .ingest into table KQL command with an abfss:// URI, which tells the Fabric Eventhouse to pull data from blob storage. Unlike ADX (which uses managed_identity=system), the Fabric path uses impersonate authentication. The storage firewall's "Allow trusted Microsoft services" bypass does not cover Fabric Eventhouse in this scenario.

Relevant code in Analytics/app.bicep:

.ingest into table ... ("abfss://ingestion@{storage}.dfs.{suffix}/...;impersonate")

For ADX, managed_identity=system is used instead, which works with the trusted services bypass.

👣 Repro steps

  1. Deploy FinOps Hub v13 with Fabric Eventhouse (fabricQueryUri set)
  2. Set enablePublicAccess = false (or manually configure storage firewall to "Selected networks" with trusted services bypass)
  3. Wait for ADF pipeline to trigger on a cost export
  4. Observe that the Fabric Eventhouse ingestion fails because it cannot read from the storage account

🤔 Expected

Ingestion should work with a firewalled storage account when using Fabric Eventhouse, the same way it works with ADX.

🔧 Environment

  • FinOps hub version: 13.0
  • Billing account type: N/A
  • Power BI report type: N/A
  • Cost Management export: N/A

ℹ️ Additional context

Reported in discussion #2058.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in Analytics/app.bicep by comparing the Fabric Eventhouse and ADX ingestion commands, then trace the ADF pipeline that issues the KQL command. Reproduce the deployment with public storage access disabled and verify that Fabric Eventhouse can ingest the parquet files from the firewalled storage account.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure
Domain
cloud, data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.