Sending over-simple DeduplicationId to FIFO type AWS SNS dropped "resolved" message

What did you do?

Configured an AWS SNS FIFO topic as a receiver to allow downstream notifications via an external system.

What did you expect to see?

SNS notifications (via an attached SQS) from alerts as they both fire and resolve, as we used send_resolved: true

What did you see instead? Under which circumstances?

This worked for most situations, but if an alert resolved within a few minutes of firing we'd get the original alert but no "resolved" message.

Environment

• System information:

Linux 5.10.205-195.804.amzn2.x86_64 x86_64

• Alertmanager version:

0.26.0

• Alertmanager configuration file:

...
    sns_configs:
      - sigv4:
          region: eu-west-1
        topic_arn: arn:aws:sns:eu-west-1:myaccount:mysns.fifo
        # If the SNS/message looks like JSON the SNS/subject must be blank or get 400 error
        subject: ""
        message: '{{ template "extras.mytemplate" .}}'
        attributes:
          # SNS attributes are forced to strings, but must never be empty
          source: alertmanager
          status: "{{.Status}}"
          num_alerts: "{{len .Alerts}}"

• Logs:

2024-01-30T12:39:24.824514000Z {"caller":"sns.go:94","integration":"sns","level":"debug","message_id":"cc112344-7d61-4432-9732-8ff6435d29a8","msg":"SNS message successfully published","sequence number":"10000000000006784000","ts":"2024-01-30T12:39:24.821Z"}
...
# 3 mins pass before it tries sending the non-identical "Resolved" message - the Status field is included in "mytemplate" so body content is different
...
2024-01-30T12:42:24.615541000Z {"caller":"sns.go:94","integration":"sns","level":"debug","message_id":"cc112344-7d61-4432-9732-8ff6435d29a8","msg":"SNS message successfully published","sequence number":"10000000000006784000","ts":"2024-01-30T12:42:24.615Z"}

Overall I think this is exactly the situation worried about in https://github.com/prometheus/alertmanager/issues/2559#issuecomment-866065698 . As we set a DeduplicationId in https://github.com/prometheus/alertmanager/blob/355c6df22924671e20acebe643dd34f77fbf0c52/notify/sns/sns.go#L158-L163 then SNS trusts it to dedupe - but the Firing and Resolved messages will have the same Group Key and lead to dropping the second message. If you left DeduplicationId blank then AWS would use a hash of the content to calc a DeduplicationId - which I think is better behaviour.

Read notify/sns/sns.go, especially the DeduplicationId handling around lines 158-163, and compare it with AWS SNS FIFO deduplication behavior. Reproduce the firing-then-resolved sequence using the configuration and logs in this issue; done means distinct firing and resolved notifications are delivered rather than one being deduplicated.