microsoft / microsoft/finops-toolkit

Harden Cost Management export retries for HTTP 429

Open
#2,257 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Tool: PowerShell
Dominant language
PowerShell
Stars
603
Forks
248
Avg merge
7d 11h
Merged PRs (30d)
11

Description

🐛 Problem

Start-FinOpsCostExport can fail instead of retrying when the Cost Management Run API returns HTTP 429. Invoke-Rest emits Write-Error for the response before the caller can reliably inspect Throttled, and the loop condition does not repeat a single-export run even when throttling is returned non-terminatingly.

The current unit test mocks Invoke-Rest returning a throttled response directly, so it does not cover the terminating error behavior observed in CI.

👣 Repro steps

  1. Run New-FinOpsCostExport -Execute or Start-FinOpsCostExport in a context where PowerShell treats Write-Error as terminating.
  2. Have the Cost Management export Run API return HTTP 429 with a Retry-After value.
  3. Observe that the command exits with an error instead of waiting and retrying.

🤔 Expected

  • HTTP 429 responses are returned to Start-FinOpsCostExport in a form it can handle.
  • The command honors the server-provided Retry-After value, with a safe fallback when absent or invalid.
  • Both single-export and multi-month runs retry the same request after throttling.
  • Retries are bounded to prevent an infinite loop.
  • Unit tests reproduce the real Invoke-Rest/Write-Error behavior and cover single-export, multi-month, retry-delay, retry-limit, and eventual-success cases.

📷 Screenshots

N/A

🔧 Environment

  • FinOps hub version: Development branch / PR #2187
  • Billing account type: N/A
  • Power BI report type: Storage-based deployment
  • Cost Management export: FOCUS cost

ℹ️ Additional context

Observed in the Hubs (manual) check for PR #2187: deployment completed, then the initial export run failed with Too many requests. Please retry after 60 seconds. (Code: 429).

Failed run: https://github.com/microsoft/finops-toolkit/actions/runs/31678558903

The two immediately preceding PR deployment runs succeeded, indicating the service-side 429 was transient while the retry-handling failure is deterministic.

🙋‍♀️ Ask for the community

We could use your help:

  1. Please vote this issue up (👍) to prioritize it.
  2. Share other Cost Management throttling responses, especially variations in status codes or retry headers.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with Start-FinOpsCostExport and its Invoke-Rest call, then inspect the existing unit test that mocks a throttled response directly. Reproduce the terminating Write-Error behavior and cover single-export, multi-month, delay, retry-limit, and eventual-success cases. Done means HTTP 429 responses are retried with Retry-After handling and bounded retries.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, powershell
Domain
api, cloud, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.