Notifications are not resilient to bad templates

What did you do?

Mis-configured a notification such that a template string was invalid and could not be rendered.

What did you expect to see?

• I expected amtool check-config to detect template errors in notifier configuration, and/or
• I expected notifiers to make a best-effort to attempt to notify despite configuration bugs, since failing to notify is a very bad outcome. A notification with incomplete data would be much more useful than no notification.

What did you see instead? Under which circumstances?

Validation with amtool didn't show any errors, and notifications were dropped.

Environment

We're running alertmanager 0.25, but I believe this issue affects all versions.

Logs look like this:

alertmanager ts=2023-08-25T20:43:03.251Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="opsgenie-receiver/opsgenie[0]: notify retry canceled due to unrecoverable error after 1 attempts: templating error: template: :1: bad character U+002D '-'"

I would be happy to write a pr to either cause amtool to check for invalid templates or to enable notifiers to send partial notifications on template errors—for example, dropping fields that fail to render. Or if there's a better approach, let me know!

Start by reproducing the invalid-template case with amtool check-config and the notifier path described in the issue; the log points to dispatch.go:352. Read the surrounding notification and template-validation flow, then clarify whether done means detecting invalid templates, preserving partial notifications, or another agreed behavior before implementing tests.