microsoft / microsoft/finops-toolkit

[Hubs] Fail ingestion when older than the retention period

Open
#1,372 0 comments 3 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Skill: Data factory Skill: Documentation Tool: FinOps hubs Type: Feature 💎
Dominant language
PowerShell
Stars
603
Forks
248
Avg merge
7d 11h
Merged PRs (30d)
11

Description

📝 Scenario

As a FinOps hub admin, I need to fail data ingestion when a month that is older than the data retention period is attempted to be ingested in order to clarify why ingested data is not available

💎 Solution

In the ingestion_ExecuteETL pipeline, check if the ingested month is older than the final data retention. If older, then fail the pipeline run with a clear error that states the data is older than the data retention and add documentation to indicate that they need to update the final retention period.

📋 Tasks

### Required tasks
- [ ] Update ingestion_ExecuteETL to fail if ingested data is older than configured retention
- [ ] Document the error code and how to resolve it
- [ ] Document the scenario in the troubleshooting guide
### Stretch goals
- [ ] TODO: Add tasks

🙋‍♀️ Ask for the community

We could use your help:

  1. Please vote this issue up (👍) to prioritize it.
  2. Leave comments to help us solidify the vision.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the ingestion_ExecuteETL pipeline and the configured final data retention setting, then inspect the existing ingestion error handling and troubleshooting guide. The work is done when older months fail with a clear error, the error code and retention-period resolution are documented, and the scenario appears in the troubleshooting guide.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, powershell
Domain
cloud, data-engineering, documentation
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.