Azure / Azure/AKS

[Feature] Native Resource Exclusions for AKS Recommended Prometheus Alerts (Label-Based Opt-Out)

Open
#5,915 1 comment 1 reaction 1 assignee Claimed by @vishiy View on GitHub
addon/ama-metrics feature-request monitoring
Dominant language
TypeScript
Stars
2.1k
Forks
395
Avg merge
2d 22h
Merged PRs (30d)
13

Description

**Is your feature request related to a problem? Please describe.**
Yes. Azure recommended AKS Prometheus alerts currently do not provide a built-in, first-class exclusion behavior for specific Kubernetes resources.
In real operations, teams often need to suppress known-noisy workloads, non-production namespaces, or specific PVC scenarios without disabling recommended alerts globally or forking all alert rules.

Today, to get this behavior, operators must:
- Clone or override recommended alert rule groups
- Add custom PromQL exclusion clauses
- Ensure kube-state-metrics label collection is configured correctly
- Maintain these customizations over time as Microsoft updates baseline recommended rules

This creates drift, adds operational risk, and makes recommended alerts harder to adopt at scale.

**Describe the solution you'd like**
Add native exclusion support to Azure recommended AKS alerts, so customers can safely opt out specific resources without replacing the full rule set.

Requested behavior:
- Support a standard exclusion label convention for Kubernetes objects, for example:
- key: excluded_from_alerts
- value: true
- Apply this across relevant recommended alert scopes where dimensions exist:
- pod-scoped alerts
- namespace-scoped alerts
- PVC-scoped alerts
- Expose exclusion behavior as a supported configuration option in Azure:
- AKS/Monitor onboarding settings
- CLI and ARM/Bicep
- Portal experience
- Ensure Microsoft-managed recommended rule updates preserve this exclusion behavior automatically.
- Document exactly which recommended alerts support exclusions and at which scope.

**Describe alternatives you've considered**
- Customizing Prometheus rule groups manually with PromQL unless clauses.
- Deploying and maintaining custom rule templates instead of Microsoft recommended defaults.
- Disabling noisy alerts entirely, which reduces coverage and signal quality.
- Namespace-level alert routing/suppression downstream in action handling systems rather than at rule-evaluation level.

All alternatives increase maintenance burden and can break when recommended rule baselines evolve.

**Additional context**
This is especially valuable for enterprise AKS environments with:
- Shared clusters (prod and non-prod workloads)
- Temporary test workloads
- Platform teams managing many clusters with centralized standards

A native exclusion capability would improve adoption of recommended alerts, reduce customization drift, and preserve signal quality while still allowing practical suppression for known exceptions.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.