Investigation: Approved SPs fraction failing by check type alerts

## Description

The **"Approved SPs fraction failing by check type (sev2.5: >= 50% of SPs failing)"** chart on the Dealbot Operational Dashboard fires frequently and auto-resolves shortly after, without a corresponding real SP-side outage.

- Chart: [`Approved SPs fraction failing by check type`](https://telemetry.betterstack.com/team/t468215/dashboards/618457/charts/12253190876) (id `12253190876`)
- Dashboard: Dealbot Operational Dashboard (id `618457`), source **Infra Prod**, mainnet only
- Backed by **4 separate threshold alerts**, one per check type, all `>= 50%` of approved SPs failing:
- `2241582239` — dataStorageStatus
- `2241582240` — retrievalStatus
- `2241582241` — dataSetChallengeStatus
- `2715738984` — sampledPieceRetrievalStatus
- All 4 share the same config: `confirmation_period=300s`, `recovery_period=1800s`, notification=email, escalation=current_team.
- The 30-minute recovery period matching the frequent auto-resolve pattern is itself a hint these are transient blips, not sustained failures.

## Investigation so far

One instance of this pattern (alert `2715738984`, sampledPieceRetrievalStatus) was investigated on 2026-07-09 (~12:45–13:35 UTC) and confirmed as a **false positive**, not a real retrieval outage:

1. Only 2–3 approved mainnet SPs generate volume for that check type — the fleet-level query only requires `active_sps >= 2` before evaluating the failing ratio, so a single misclassified SP is enough to cross the 50% threshold.
2. Per-SP `is_failing` classification comes purely from a rolling 1h delta of a `maxMerge(value_max)` counter, not raw event outcomes: a provider counts as failing when `t >= 5 AND s/t < 0.7` in that window.
3. Cross-checking raw dealbot-worker pod logs for the flagged provider showed genuine successful retrievals landing *during* the alert window that did not increment the counter for ~100 minutes, while one small/fast failure did land immediately — so a provider that was actually succeeding read as 100% failing.
4. Fleet-wide `jobs_completed_total` for retrieval_sampled climbed continuously through the window, confirming the underlying check pipeline was healthy the whole time. The alert was purely a metric-export/aggregation-lag artifact.

This chart's query has no floor on absolute event volume or on `failing_sps` count — only `active_sps >= 2` — so the same mechanism (tiny approved-SP fleet + counter lag) can plausibly trip any of the other 3 alerts too, not just sampledPieceRetrievalStatus.