Costs_final_v1_2 loses extents when PropagateIngestionProperties=true

Description

Environment

• FinOps Toolkit version: 12.0

• ADX update policy:

  • Source: Costs_raw
  • Query: Costs_transform_v1_2()
  • IsTransactional: true
  • PropagateIngestionProperties: true

Problem

After ingesting multiple parquet files into Costs_raw, only some extents appear in Costs_final_v1_2.

Example:

Costs_raw

March export:

• part_0_0001 – 270 rows
• part_1_0001 – 329,262 rows
• part_2_0001 – 355,983 rows
• part_3_0001 – 356,014 rows
• part_4_0001 – 196,431 rows

Total rows:

1,237,960

Costs_final_v1_2

Only the last extent was present:

• part_4_0001 – 196,431 rows

The same behavior occurred for another export where only the last part survived.

No update policy failures or ingestion failures were reported.

Investigation performed

• Verified Costs_raw extents were healthy.
• Verified Costs_transform_v1_2() returns correct results when executed manually.
• Verified row counts and total cost values.
• Verified there were no update policy failures.
• Verified there were no ingestion failures.
• Tables were recreated from scratch.

Root cause

The issue appears to be caused by:

"PropagateIngestionProperties": true

with propagated drop-by: tags.

Workaround

Changing the update policy to:

{
  "IsEnabled": true,
  "Source": "Costs_raw",
  "Query": "Costs_transform_v1_2()",
  "IsTransactional": true,
  "PropagateIngestionProperties": false
}

resolved the issue.

After changing the setting:

• Costs_raw row count = 1,237,960
• Costs_final_v1_2 row count = 1,237,960

Total cost values also matched.