microsoft / microsoft/finops-toolkit
[Hubs] Archive data – Change ADF to move parquet to archive/raw container
Nobody has claimed this yet.
- Dominant language
- PowerShell
- Stars
- 603
- Forks
- 248
- Avg merge
- 7d 11h
- Merged PRs (30d)
- 11
Description
📝 Scenario
As a FinOps practitioner, I need to move raw data to cold storage in order to minimize storage costs for data that is not accessed frequently
💎 Solution
Add an option to move raw data to cold or archive storage after it's ingested into Azure Data Explorer.
ℹ️ Additional context
Add a boolean flag to the config/settings.json file and check that during ingestion to determine whether data should be archived or not.
🙋♀️ Ask for the community
We could use your help:
- Please vote this issue up (👍) to prioritize it.
- Leave comments to help us solidify the vision.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with config/settings.json and the Azure Data Factory ingestion workflow mentioned in the issue. Trace where ingestion into Azure Data Explorer is configured, then determine how the new archive setting should control moving raw data to the archive/raw container. Done means the setting is honored and raw data is archived only when enabled.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, powershell
- Domain
- cloud, data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 32/100