Feature: Multi-Cloud FOCUS Test Data Generator Script

🚀 Feature: Multi-Cloud FOCUS Test Data Generator Script

Summary

Add a PowerShell script (Generate-MultiCloudTestData.ps1) that generates synthetic, multi-cloud, FOCUS-compliant cost data for testing and validating FinOps Hub deployments. This script was originally developed for the FinOps Hub AVM module and has been validated against production FinOps Hub deployments with 498K+ rows of data.

Problem

Today, there is no easy way to test a FinOps Hub deployment end-to-end without connecting it to a real Azure Cost Management export. This creates several challenges:

1. Testing new deployments — Users must wait for a real export run (which requires billing data) before they can validate their hub is working.
2. Multi-cloud scenarios — There's no way to test AWS, GCP, or DataCenter ingestion without real data from those providers.
3. Dashboard validation — Dashboard KQL queries rely on specific column combinations (commitment discounts, Azure Hybrid Benefit, marketplace, SKU details) that are hard to manually construct.
4. CI/E2E testing — Automated tests need representative data to validate ingestion pipelines and ADX schema setup.
5. Demo environments — Creating convincing demo data for presentations and training requires significant manual effort.

Proposed Solution

A self-contained PowerShell script that generates realistic synthetic cost data following the FOCUS specification (v1.0–1.3). The script:

Core Capabilities

• 4 cloud providers: Azure, AWS, GCP, and DataCenter (on-premises)
• FOCUS 1.0–1.3 compliant: All mandatory + conditional FOCUS columns
• Dashboard-aligned: Every column referenced by FinOps Hub dashboard KQL queries is populated with realistic values
• Persistent identities: Resources, billing accounts, and subscriptions are consistent across all days (enabling realistic trending)
• Budget scaling: Generated costs are scaled to a target budget (default $500K)
• Memory-safe streaming: Rows are streamed to CSV daily, avoiding OOM on large datasets

Data Realism Features

Output & Upload

• Formats: Parquet (via Python/pyarrow), CSV, or both
• Azure Storage upload: Uploads to msexports (Azure) and ingestion (AWS/GCP/DC) containers with proper folder structure and manifest.json files
• ADF trigger management: Optionally starts ADF triggers before upload to ensure BlobCreated events are captured

Usage Examples

# Basic: 6 months, all providers, 500K rows, $500K budget
.\Generate-MultiCloudTestData.ps1

# Quick test: 3 months, 100K rows
.\Generate-MultiCloudTestData.ps1 -MonthsOfData 3 -TotalRowTarget 100000 -TotalBudget 50000

# Full pipeline: generate + upload + trigger ADF
.\Generate-MultiCloudTestData.ps1 -Upload -StorageAccountName "stfinopshub" -ResourceGroupName "rg-finopshub" -AdfName "adf-finopshub" -StartTriggers

# Single provider
.\Generate-MultiCloudTestData.ps1 -CloudProvider Azure -TotalRowTarget 200000

Prerequisites

• PowerShell 7+
• Python 3 with pandas and pyarrow (for Parquet output)
• Azure CLI (for -Upload functionality)

Suggested Location

src/scripts/Generate-MultiCloudTestData.ps1 or src/templates/finops-hub/test/Generate-MultiCloudTestData.ps1

Additional Context

• Script is ~1,430 lines of pure PowerShell (data generation) + Python (Parquet conversion)
• Has been tested with 498K rows of data successfully ingested into a FinOps Hub with ADX
• Generates data that exercises all FinOps Hub dashboard pages: Cost Summary, Commitment Discounts, Invoicing, Data Quality, SKU Analysis
• Row distribution for "All" providers: ~60% Azure, ~20% AWS, ~15% GCP, ~5% DataCenter

Start by reviewing the proposed locations, src/scripts/Generate-MultiCloudTestData.ps1 or src/templates/finops-hub/test/Generate-MultiCloudTestData.ps1, along with the PowerShell 7, Python, pyarrow, and Azure CLI prerequisites. Done means a self-contained generator produces the described multi-cloud FOCUS data, supports the documented output and upload options, and exercises the listed FinOps Hub scenarios.