microsoft / microsoft/finops-toolkit

[ADF] msexports_ETL_ingestion fails with PathNotFound for reservation recommendation exports

Open
#2,088 2 comments 0 reactions 0 assignees View on GitHub

A pull request for this has already been merged.

  • #2247 by @flanakin — merged
Skill: Data factory Tool: FinOps hubs Type: Cleanup
Dominant language
PowerShell
Stars
603
Forks
248
Avg merge
7d 11h
Merged PRs (30d)
11

Description

Bug description

The msexports_ETL_ingestion pipeline fails with a PathNotFound error when processing reservation recommendation exports. The pipeline tries to read Parquet files from a path like:

ingestion/Recommendations/<year>/<month>/providers/microsoft.billing/billingaccounts/<id>/billingprofiles/<id>-reservationrecommendations-daily

The path does not exist in the storage account, causing the Get Existing Parquet Files activity to fail with:

ADLS Gen2 operation failed for: Operation returned an invalid status code 'NotFound'.
ErrorCode: 'PathNotFound'. Message: 'The specified path does not exist.'

This cascades into msexports_ExecuteETL and subsequently ingestion_ExecuteETL / ingestion_ETL_dataExplorer failures.

Steps to reproduce

  1. Deploy a FinOps hub with Fabric and recommendations enabled
  2. Have a Cost Management export configured for reservation recommendations
  3. Wait for the daily trigger to run the msexports_ExecuteETL pipeline

Expected behavior

The pipeline should handle the case where the ingestion path does not yet exist (e.g., first run or no data from the export) gracefully, instead of failing with PathNotFound.

Environment

  • FinOps toolkit version: 12 (deployed), 14.0-dev (source)
  • Analytics backend: Microsoft Fabric

🤖 Generated with Claude Code

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Review the linked pull request #2247 and the msexports_ETL_ingestion pipeline, especially the Get Existing Parquet Files activity and its PathNotFound failure. Reproduce with a reservation recommendations export that has no ingestion path yet, then verify that msexports_ExecuteETL and downstream ingestion pipelines complete gracefully when the path is absent.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure
Domain
cloud, data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.