[ROSAENG-62134] ocm-agent fleet mode: restoreNotificationStatus causes infinite retry storm on 429 rate limit

Summary

When the OCM service log API returns HTTP 429 (rate limit exceeded), ocm-agent fleet mode enters an infinite retry loop. The restoreNotificationStatus() function rolls back lastTransitionTime after every failed send, causing canSendNotification() to return true on every reconciliation cycle (~15 min). This generates ~24 wasted API requests per hour, per affected cluster, indefinitely.

Root Cause

The issue is in pkg/handlers/webhookrhobsreceiver.go in the processAlert function (line 415-423). The code flow:

updateNotificationStatus() sets lastTransitionTime = now and increments FiringNotificationSentCount (written to K8s MFNR)

sendNotification() POSTs to OCM API → receives 429

restoreNotificationStatus() rolls back both lastTransitionTime and FiringNotificationSentCount to their pre-update values

On the next reconciliation (~15 min later), canSendNotification() reads the rolled-back lastTransitionTime, finds it is older than ResendWait (24h), and returns true

Steps 1-4 repeat indefinitely

The rollback was designed as a workaround for the optimistic update pattern (status is updated before the send, so it must be rolled back on failure). However, it does not distinguish between transient errors (where retry is appropriate) and rate-limit errors (where retry makes the problem worse).

Impact

API waste: Each 15-min cycle fires 1 initial POST + 2 SDK retries per cluster = 3 requests. With 2 clusters affected, that is 6 requests/cycle or ~24 requests/hour, all returning 429.

Envoy/Limitador pressure: The fleet service account's request-rate quota (enforced by the Envoy sidecar via Limitador at limitador.app-sre-rate-limiting.svc:8081) is consumed by these wasted retries, potentially starving legitimate service log sends for other clusters on the same management cluster.

Silent service log loss: The service log is never created — the customer never gets notified — but the system keeps trying without success.

Observed Behavior

Error in ocm-agent-fleet logs:

error="can't post service log: status is 429, identifier is '8', code is 'OCM-CA-8' and operation identifier is '...': Exceeds rate limit request"

Proposed Fix

Two changes:

1. pkg/ocm/ocm.go — Return typed RateLimitError on 429

Add a RateLimitError type and return it from SendServiceLog when the response status is 429. This allows callers to distinguish rate limiting from other failures.

2. pkg/handlers/webhookrhobsreceiver.go — Rate-limit-aware backoff

Start in pkg/ocm/ocm.go at SendServiceLog and in pkg/handlers/webhookrhobsreceiver.go at processAlert and restoreNotificationStatus. Trace the 429 response through the status update and rollback flow, then verify that rate-limited sends no longer create an infinite retry cycle while other failures retain their existing behavior.