microsoft / microsoft/finops-toolkit

[Hubs] Re-ingest costs when prices are ingested

Open
#1,734 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Tool: FinOps hubs Type: Feature 💎
Dominant language
PowerShell
Stars
603
Forks
248
Avg merge
7d 11h
Merged PRs (30d)
11

Description

📝 Scenario

As a FinOps hub admin, I need to re-ingest costs when prices are updated for the month in order to populate missing price data from Cost Management exports

💎 Solution

Update the ingestion_ExecuteETL pipeline to do the following:

  1. After the 'If Has Capacity' activity runs successfully, check if the current export is for the "Prices" dataset.
  2. If it is for "Prices", run a KQL extents query against the Ingestion database to see if there's an extent for the Costs dataset for the current month that is being ingested.
  3. If no Costs data has been ingested for the current month, do nothing.
  4. If Costs data for the current month has been ingested already, run the ingestion_ExecuteETL pipeline asynchronously for Costs and the current month.

🙋‍♀️ Ask for the community

We could use your help:

  1. Please vote this issue up (👍) to prioritize it.
  2. Leave comments to help us solidify the vision.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the ingestion_ExecuteETL pipeline and trace the successful 'If Has Capacity' path and the Prices and Costs dataset inputs. Review the KQL extents query against the Ingestion database and how asynchronous pipeline runs are configured. Done means Prices ingestion triggers a Costs run only when current-month Costs data already exists, and otherwise does nothing.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure
Domain
cloud, data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.