microsoft / microsoft/finops-toolkit
[Hubs] Fail ingestion when older than the retention period
Nobody has claimed this yet.
- Dominant language
- PowerShell
- Stars
- 603
- Forks
- 248
- Avg merge
- 7d 11h
- Merged PRs (30d)
- 11
Description
📝 Scenario
As a FinOps hub admin, I need to fail data ingestion when a month that is older than the data retention period is attempted to be ingested in order to clarify why ingested data is not available
💎 Solution
In the ingestion_ExecuteETL pipeline, check if the ingested month is older than the final data retention. If older, then fail the pipeline run with a clear error that states the data is older than the data retention and add documentation to indicate that they need to update the final retention period.
📋 Tasks
### Required tasks
- [ ] Update ingestion_ExecuteETL to fail if ingested data is older than configured retention
- [ ] Document the error code and how to resolve it
- [ ] Document the scenario in the troubleshooting guide
### Stretch goals
- [ ] TODO: Add tasks
🙋♀️ Ask for the community
We could use your help:
- Please vote this issue up (👍) to prioritize it.
- Leave comments to help us solidify the vision.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the ingestion_ExecuteETL pipeline and the configured final data retention setting, then inspect the existing ingestion error handling and troubleshooting guide. The work is done when older months fail with a clear error, the error code and retention-period resolution are documented, and the scenario appears in the troubleshooting guide.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, powershell
- Domain
- cloud, data-engineering, documentation
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100