Alert Rules Labels do not appear in Notification Email when being used in annotations/description prometheus field

What did you do?
Labels that are defined in the alert/rule and then are used annotation/description do not appear in email notifications

Alert:

As you can see all other labels except namespace and alertname are not working

What did you expect to see?
I expect for those labels to appear in the email notifications

What did you see instead? Under which circumstances?
When the labels are defined in the alert rule and then are used in annotation/descriptions do not appear in email notifications.

Environment

• System information:

Kubernetes 1.24
For the alertmanager deployment we use helm chart Kube-Prometheus-Stack:42.0.3 which use the following image from alertmanager: https://quay.io/prometheus/alertmanager:v0.24.0

• Alertmanager version:

Alertmanager: https://quay.io/prometheus/alertmanager:v0.24.0

• Prometheus version:

Prometheus: https://quay.io/prometheus/prometheus:v2.39.1

• Alertmanager configuration file:

alertmanagerSpec = {

      priorityClassName = var.priority_class

      nodeSelector = local.stateful_node_group_config

      # See: https://github.com/helm/charts/issues/15275
      logFormat = "logfmt"

      image = {
        registry   = local.registry
        repository = local.images.alertmanager.main
      }

      resources = {
        limits = {
          cpu    = "50m"
          memory = "100Mi"
        }
        requests = {
          cpu    = "30m"
          memory = "50Mi"
        }
      }
    }

    templateFiles = {
      "sns_template.tmpl" = <<EOF
{{ define "sns.default.message" }}{{ "{" }}"receiver": "{{ .Receiver }}","status": "{{ .Status }}","alerts": [{{ range $alertIndex, $alerts := .Alerts }}{{ if $alertIndex }}, {{ end }}{{ "{" }}"status": "{{ $alerts.Status }}"{{ if gt (len $alerts.Labels.SortedPairs) 0 -}},"labels": {{ "{" }}{{ range $index, $label := $alerts.Labels.SortedPairs }}{{ if $index }}, {{ end }}"{{ $label.Name }}": "{{ $label.Value }}"{{ end }}{{ "}" }}{{- end }}{{ if gt (len $alerts.Annotations.SortedPairs ) 0 -}},"annotations": {{ "{" }}{{ range $index, $annotations := $alerts.Annotations.SortedPairs }}{{ if $index }}, {{ end }}"{{ $annotations.Name }}": "{{ $annotations.Value | reReplaceAll "\n" "\\n" }}"{{ end }}{{ "}" }}{{- end }},"startsAt": "{{ $alerts.StartsAt }}","endsAt": "{{ $alerts.EndsAt }}","generatorURL": "{{ $alerts.GeneratorURL }}","fingerprint": "{{ $alerts.Fingerprint }}"{{ "}" }}{{ end }}]{{ if gt (len .GroupLabels) 0 -}},"groupLabels": {{ "{" }}{{ range $index, $groupLabels := .GroupLabels.SortedPairs }}{{ if $index }}, {{ end }}"{{ $groupLabels.Name }}": "{{ $groupLabels.Value }}"{{ end }}{{ "}" }}{{- end }}{{ if gt (len .CommonLabels) 0 -}},"commonLabels": {{ "{" }}{{ range $index, $commonLabels := .CommonLabels.SortedPairs }}{{ if $index }}, {{ end }}"{{ $commonLabels.Name }}": "{{ $commonLabels.Value }}"{{ end }}{{ "}" }}{{- end }}{{ if gt (len .CommonAnnotations) 0 -}},"commonAnnotations": {{ "{" }}{{ range $index, $commonAnnotations := .CommonAnnotations.SortedPairs }}{{ if $index }}, {{ end }}"{{ $commonAnnotations.Name }}": "{{ $commonAnnotations.Value | reReplaceAll "\n" "\\n" }}"{{ end }}{{ "}" }}{{- end }}{{ "}" }}{{ end }}
{{ define "sns.default.subject" }}[{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}]{{ end }}
EOF
    }

    config = {
      global = {
        resolve_timeout = "5m"
      }
      route = {
        group_by = [
          "job",
          "severity"
        ]
        group_wait      = "30s"
        group_interval  = "5m"
        repeat_interval = "12h"
        receiver        = "null"
        routes = concat(
          [
            for alertname in local.alertname_blacklist :
            {
              match = {
                alertname = alertname
              }
              receiver = "null"
            }
          ],
          var.platform_upgrade_ongoing ? [
            for alertname in local.alertname_blacklist_platform_upgrade_ongoing :
            {
              match = {
                alertname = alertname
              }
              receiver = "null"
            }
          ] : [],
          [
            for receiver in var.teams_alerting_channels :
            {
              match_re = {
                teams_channel_name = receiver.name
              }
              receiver   = receiver.name
              group_wait = "10s"
              group_by   = ["namespace", "mc_id", "event_id", "priority"]
              continue   = true
            }
          ],
          [
            {
              match_re = {
                namespace : "^\\S+[^ ]$"
                mc_id : "^\\S+[^ ]$"
                event_id : "^\\S+[^ ]$"
                priority : "^\\S+[^ ]$"
              }
              receiver = "itsm"
              group_by = ["namespace", "mc_id", "event_id", "priority"]
            }
          ],
          [
            for alertname in local.alertname_whitelist :
            {
              match_re = {
                namespace   = "^\\s*$"
                alertname   = alertname
                environment = "prod|e2e"
              }
              receiver       = "itsm"
              group_wait     = "10m"
              group_interval = "10m"
              group_by       = ["alertname", "product", "environment", "region", "severity"]
              continue       = true
            }
          ]
        )
      }

      receivers = concat(
        [
          {
            name = "null"
          },
        ],
        [
          merge(
            {
              name = "itsm"
            },
            local.itsm_receiver_config
          )
        ],
        [
          for receiver in var.teams_alerting_channels :
          {
            name : receiver.name,
            webhook_configs : [
              merge({
                url = "http://prometheus-msteams:2000/${receiver.name}"
              }, local.alertmanager_send_resolved)
            ]
          }
        ]
      )
    }
  }

• Prometheus configuration file:

Not relevant

• Notes: These are custom alerts

Start with the supplied sns_template.tmpl and the alert rule annotations described in the issue, then reproduce the notification using Alertmanager v0.24.0. Trace how rule labels reach annotation and email rendering; done means labels referenced in annotations appear in the resulting notification email, with a regression test if the relevant test location is identified.