alertmanager: Add guidance on rate window vs "for"

I think the documentation could do a better job explaining what for is for. The current documentation says this about the for field of an alert:

The optional for clause causes Prometheus to wait for a certain duration between first encountering a new expression output vector element and counting an alert as firing for this element. In this case, Prometheus will check that the alert continues to be active during each evaluation for 10 minutes before firing the alert. Elements that are active, but not firing yet, are in the pending state.

I had situation where an alert was firing too often and I wanted to make it more sensitive. Simplified example:

  - alert: HighErrorRate
    expr: rate(errors[5m])
    for: 10m

I wanted to figure out whether I should increase the rate range ([5m]) or increase the for: 10m. After some talking to people and reading the SRE Workbook, my conclusion was that, if you are measuring a rate or calculating an average, it is better to use only the range ([5m]), and omit the for. The reasoning seems to be that this improves time-to-detection and recall.

On the other hand, it seems like the best-case use for for is on gauges and boolean-valued time series.

Is that an accurate summary? If I wrote up a PR tweaking the docs to clarify that, would that be something you'd accept?

Thanks,
Jacob

Start from the documentation passage quoted in the issue and review Prometheus alerting guidance for the interaction between a rate range and the for duration. Verify the proposed distinction against the relevant alerting semantics, then update the documentation so readers can choose between these options and understand when each is appropriate.