microsoft / microsoft/finops-toolkit

[Power BI] Follow-ups from PR #1849 ARG batching review: shared subscription-list caching and column-shape normalization

Open
#2,224 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Tool: Power BI
Dominant language
PowerShell
Stars
603
Forks
248
Avg merge
7d 11h
Merged PRs (30d)
11

Description

Tracking issue for two review findings on PR #1849 ("[Power BI] Address ARG payload size errors") that were judged valid but out of scope for that PR's minimal-diff fix. Both apply to ftk_QueryARG in src/power-bi/kql/Shared.Dataset/definition/expressions.tmdl and the mirrored src/power-bi/storage/Shared.Dataset/definition/expressions.tmdl copy.

1. Hoist the subscription-list query into one shared, cached expression

Currently every batched table calls ftk_QueryARG, which re-runs the resourcecontainers subscription-list query once per table. For a 1,000-subscription tenant across the twelve adopting tables, that's roughly 12 extra ARG calls that could be eliminated by querying the subscription list once (e.g., via a shared/cached expression or the existing Subscriptions table) and passing the result into ftk_QueryARG instead of re-deriving it per call.

Reference: https://github.com/microsoft/finops-toolkit/pull/1849#discussion_r2919445699 (comment id 3688707701)

2. Normalize combined batch results against each table's authoritative column list

ftk_QueryARG's Table.Combine(Filtered) assumes every batch returns the same column shape. Azure Resource Graph infers columns from the data actually returned, so a batch missing an mv-expanded or dynamic-typed column can come back with a different column set/type than another batch, and Table.Combine unions rather than erroring (nulls fill the gaps). Each calling table's NullHandling block already declares an authoritative column list for the empty-table case; normalizing the combined table against that same list on the success path (e.g., via Table.SelectColumns with MissingField.UseNull plus explicit reordering) would make the shape deterministic regardless of batch order/content.

This needs to be verified against live Azure Resource Graph data in Power BI Desktop before merging, since the type/column drift only manifests with real tenant data variance across batches, and 24 files (12 tables x kql/storage) would need updating.

Reference: https://github.com/microsoft/finops-toolkit/pull/1849#discussion_r2919445724 (comment id 3688707720)

Why deferred

Both are legitimate improvements but would require either changing the ftk_QueryARG shared function signature (used by 12+ tables across 2 datasets) or touching column-list logic in 24 files, and validating either change needs real Power BI Desktop + live Azure Resource Graph access that wasn't available while addressing PR #1849's other review findings. Fixing them together, with real validation, is better done as a dedicated follow-up.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with ftk_QueryARG in src/power-bi/kql/Shared.Dataset/definition/expressions.tmdl and its mirrored storage path, then inspect its 12+ callers and their NullHandling column lists. Validate changes in Power BI Desktop against live Azure Resource Graph data; done means subscription data is shared or cached and combined results have a deterministic shape across the affected tables and dataset copies.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure
Domain
analytics, cloud, data
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.