microsoft / microsoft/finops-toolkit
[ADF] msexports_ETL_ingestion fails with PathNotFound for reservation recommendation exports
A pull request for this has already been merged.
- #2247 by @flanakin — merged
- Dominant language
- PowerShell
- Stars
- 603
- Forks
- 248
- Avg merge
- 7d 11h
- Merged PRs (30d)
- 11
Description
Bug description
The msexports_ETL_ingestion pipeline fails with a PathNotFound error when processing reservation recommendation exports. The pipeline tries to read Parquet files from a path like:
ingestion/Recommendations/<year>/<month>/providers/microsoft.billing/billingaccounts/<id>/billingprofiles/<id>-reservationrecommendations-daily
The path does not exist in the storage account, causing the Get Existing Parquet Files activity to fail with:
ADLS Gen2 operation failed for: Operation returned an invalid status code 'NotFound'.
ErrorCode: 'PathNotFound'. Message: 'The specified path does not exist.'
This cascades into msexports_ExecuteETL and subsequently ingestion_ExecuteETL / ingestion_ETL_dataExplorer failures.
Steps to reproduce
- Deploy a FinOps hub with Fabric and recommendations enabled
- Have a Cost Management export configured for reservation recommendations
- Wait for the daily trigger to run the
msexports_ExecuteETLpipeline
Expected behavior
The pipeline should handle the case where the ingestion path does not yet exist (e.g., first run or no data from the export) gracefully, instead of failing with PathNotFound.
Environment
- FinOps toolkit version: 12 (deployed), 14.0-dev (source)
- Analytics backend: Microsoft Fabric
🤖 Generated with Claude Code
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Review the linked pull request #2247 and the msexports_ETL_ingestion pipeline, especially the Get Existing Parquet Files activity and its PathNotFound failure. Reproduce with a reservation recommendations export that has no ingestion path yet, then verify that msexports_ExecuteETL and downstream ingestion pipelines complete gracefully when the path is absent.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure
- Domain
- cloud, data-engineering
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100