microsoft / microsoft/AzureMonitorCommunity

Shared ALZ logs cost - query that runs too long

Open
#91 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
PowerShell
Stars
1.2k
Forks
493
PR merge metrics
No merged PRs in 30d

Description

In a shared Azure Landing Zone infrastructure, we are running a dozen or so SW applications, and we must report the cost of LAW ingestion of all the Azure resources allocated to each application (we call them "Outcomes") in this environment. We built a workbook and in it fashioned some queries, one of them "by outcome" and it takes a loooooong time to complete (I have never seen it complete) based on several TiB of data ingested over the last 30 days.

Screenshot_Outcome_query_edit1
Screenshot_Outcome_query_runs

The Kusto Query

Parameter Query :- (Scoped to Subscription)

ResourceContainers
| where type=='microsoft.resources/subscriptions/resourcegroups'
| extend Tag = todynamic(tags)
| extend TeamName = Tag["TEAM NAME"]
| where isnotempty(TeamName)
| project Owner = strcat("'",name,'#',tostring(TeamName),"'")

Actual Query :- (Scoped to Log Analytics Workspace)

let OutcomeTable = datatable(ResourceGroupOwner:string) {ResourceGroupOwnerList};
find where TimeGenerated {TimeRange:value} project _ResourceId, _BilledSize, _IsBillable, TimeGenerated
| where _IsBillable == true
| extend ResourceGroup = case(isempty(_ResourceId),"Infrastructure",tostring(split(_ResourceId, '/')[4]))
| where isnotempty(ResourceGroup)
| summarize IngestedData = sum(_BilledSize) by ResourceGroup
| join kind=leftouter (OutcomeTable | extend ResourceGroup = tostring(split(ResourceGroupOwner,'#')[0]), Owner = tostring(split(ResourceGroupOwner,'#')[1]) | project ResourceGroup, Owner) on ResourceGroup
| project Owner = case(isempty(Owner),ResourceGroup,Owner), IngestedData
| summarize sum(IngestedData) by Owner

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the workbook's "by outcome" Kusto query and its ResourceGroupOwner parameter query, then run it against the stated 30-day Log Analytics range. Check the query's execution behavior and aggregation steps; done means the query completes and reports billable ingestion by Owner for the shared Azure resources.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure
Domain
cloud, observability-sre
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.