Debug Azure Load Balancer Alerts

Context

The Azure Load Balancer alert rules (azure_lb_health_probe_down, azure_lb_data_path_down, azure_lb_snat_port_exhaustion) in azure_loadbalancer.yaml have been temporarily removed because the underlying metrics never appear in Grafana. The alerts were permanently firing in NoData state.

The Alloy prometheus.exporter.azure component successfully discovers load balancer resources via Azure Resource Graph, but silently fails to fetch metrics from Azure Monitor — it makes zero API calls for Microsoft.Network/loadBalancers while other resource types (NetApp, PostgreSQL, Storage) work correctly with identical configuration patterns.

Investigation summary

Two debugging sessions confirmed:

• Resource discovery works: Alloy finds the load balancers and reports 1 scrape target
• Metrics exist in Azure: az monitor metrics list returns data for UsedSnatPorts, AllocatedSnatPorts, and SnatConnectionCount
• No Azure Monitor API calls: Alloy's azurerm_api_request_count shows 0 requests for microsoft.network, vs 125+ for microsoft.netapp
• No errors logged: The exporter fails silently with no warnings or errors
• Same identity/permissions: All Azure exporters use the same managed identity and subscription

Configuration variations tested without success: case-insensitive resource group filters (==, =~, tolower()), removing included_dimensions, adding explicit metric_aggregations, removing metrics that had no data (DipAvailability, VipAvailability).

What needs to be done

1. Diagnose the Alloy exporter failure — likely candidates:
   • Alloy bug specific to Microsoft.Network/loadBalancers resource type
   • Load balancer metrics requiring a metric_namespace parameter or explicit dimension filters
   • Silent validation failure during metric collection setup
2. Re-enable alerts once metrics flow into Grafana — restore alert definitions from git history and remove the deleteRules block in azure_loadbalancer.yaml

Suggested next steps

• Try adding metric_namespace = "Microsoft.Network/loadBalancers" to the Alloy exporter config
• Enable Alloy debug logging (logging { level = "debug" }) to capture internal exporter behavior
• Filter to a single load balancer (name == 'kubernetes') to isolate multi-resource issues
• If none of the above work, file a bug against grafana/alloy

Related files

• python-pulumi/src/ptd/grafana_alerts/azure_loadbalancer.yaml — currently contains deleteRules to clean up the alerts
• python-pulumi/src/ptd/pulumi_resources/grafana_alloy.py — Alloy exporter configuration

Start with python-pulumi/src/ptd/pulumi_resources/grafana_alloy.py and python-pulumi/src/ptd/grafana_alerts/azure_loadbalancer.yaml; enable Alloy debug logging and test a single load balancer using the suggested metric namespace and filters. Compare Azure API request counts and Grafana metrics with working exporters. Done means the load-balancer metrics flow and the alert definitions are restored without NoData firing.