microsoft / microsoft/finops-toolkit
[Hubs] Re-ingest costs when prices are ingested
Nobody has claimed this yet.
- Dominant language
- PowerShell
- Stars
- 603
- Forks
- 248
- Avg merge
- 7d 11h
- Merged PRs (30d)
- 11
Description
📝 Scenario
As a FinOps hub admin, I need to re-ingest costs when prices are updated for the month in order to populate missing price data from Cost Management exports
💎 Solution
Update the ingestion_ExecuteETL pipeline to do the following:
- After the 'If Has Capacity' activity runs successfully, check if the current export is for the "Prices" dataset.
- If it is for "Prices", run a KQL extents query against the Ingestion database to see if there's an extent for the Costs dataset for the current month that is being ingested.
- If no Costs data has been ingested for the current month, do nothing.
- If Costs data for the current month has been ingested already, run the ingestion_ExecuteETL pipeline asynchronously for Costs and the current month.
🙋♀️ Ask for the community
We could use your help:
- Please vote this issue up (👍) to prioritize it.
- Leave comments to help us solidify the vision.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at the ingestion_ExecuteETL pipeline and trace the successful 'If Has Capacity' path and the Prices and Costs dataset inputs. Review the KQL extents query against the Ingestion database and how asynchronous pipeline runs are configured. Done means Prices ingestion triggers a Costs run only when current-month Costs data already exists, and otherwise does nothing.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure
- Domain
- cloud, data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100