Failed alerts retry forever

## Expected Behavior

## Current Behavior

When an alert fails to run (timeout etc) it will try again in 5 seconds, but if it keeps failing it will stay in that loop forever apparently. If this is a large query, it can then cause other alerts to start to timeout, and the problem just snowballs.
This is logged in server.log, however, there is nothing to tell the user this is happening, so if it was a critical alert you would have no idea that it was no longer working properly, and this failure could go on for days unnoticed.

## Possible Solution

There should be a way to control this behavior, ideally at the individual alert level because some alerts you may want to retry a little more than others depending on criticality etc. or have a global configurable max number of retries etc.

## Steps to Reproduce (for bugs)

1.
2.
3.
4.

## Context

## Your Environment

* Graylog Version: 6.1.5
* Java Version:
* OpenSearch Version:
* MongoDB Version:
* Operating System:
* Browser version: