Brownout detection: back off checks against consistently failing SPs
- Dominant language
- TypeScript
- Stars
- 3
- Forks
- 9
- Avg merge
- 4d 14h
- Merged PRs (30d)
- 6
Description
Dealbot checks every active SP at a fixed rate regardless of how that SP is performing. In [production](https://github.com/FilOzone/dealbot/blob/main/docs/checks/production-configuration-and-approval-methodology.md) that is `DEALS_PER_SP_PER_HOUR`=4 and `RETRIEVALS_PER_SP_PER_HOUR`=4 (96/day each) plus `PULL_CHECKS_PER_SP_PER_HOUR`=1 (24/day). An SP failing every check gets the same ~216 checks/day, each data storage check carrying a 10MB upload, as an SP passing every check.
The only lever we have today is manual and binary: `_BLOCKED_SP_IDS` / `_BLOCKED_SP_ADDRESSES`, read by `isSpBlocked()` in [`common/sp-blocklist.ts`](https://github.com/FilOzone/dealbot/blob/main/apps/backend/src/common/sp-blocklist.ts) when schedules are built in `ensureScheduleRows()` ([`jobs/jobs.service.ts`](https://github.com/FilOzone/dealbot/blob/main/apps/backend/src/jobs/jobs.service.ts#L1310)). There is no per-SP backoff, cooldown, or circuit breaker. The exponential backoff already in the codebase is for subgraph and RPC fetches, not for SP checks.
SP operators have asked for the missing middle: a way for dealbot to notice an SP is in a brownout and stop battering it, without a human editing an env var and redeploying.
## Why Important
An SP in a degraded state keeps absorbing full check load from us while it is trying to recover. On our side, checks against a known-broken SP spend worker capacity and RPC budget to re-learn a failure we already recorded, and provisioning against a struggling SP has cost us money before (#402).
## Done Criteria
This is a design decision, not an agreed implementation. Done when we have a written proposal that answers the open questions below and a go/no-go on building it. If it is a go, file the implementation issues and link them here.
Open questions, roughly in the order that would kill the idea fastest:
1. Is dealbot's check volume actually a meaningful share of a struggling SP's load? If we are a rounding error next to their client traffic, backing off makes their dealbot graph look calmer without helping them recover, and the real fix is elsewhere. We should get this number before building anything.
2. How does backing off interact with the [approval thresholds](https://github.com/FilOzone/dealbot/blob/main/docs/checks/production-configuration-and-approval-methodology.md#approval-acceptance-criteria)? They need ≥97% over a minimum of 200 checks. Fewer checks means slower sample accumulation, so a brownout SP is slower both to drop below the threshold and to prove it recovered. Do checks during a brownout still count toward the sample window?
3. Does an automatic "fail more, get checked less" rule create an incentive to fail? It cannot be usable to dodge sampling ahead of an approval decision.
4. What triggers a brownout and what exits it? Consecutive failures vs. a rolling success rate, and a probe-rate floor low enough to be gentle but high enough that recovery is detected on its own.
5. How do we attribute failure before penalizing an SP? Dealbot-side faults (our own bugs, RPC timeouts per #603) must not trigger a brownout against the SP.
## User/Customer
SPs being checked by dealbot, and SP approvers reading the resulting data.
## Notes
Distinct from the neighbors, and should compose with them rather than replace them:
- #280 (automated approve/unapprove alerting) changes an SP's status. A brownout is a temporary rate change with no status change.
- #440 (programmatic maintenance windows) is SP-declared and scheduled. A brownout is dealbot-detected and unplanned.
If we build it, the levers already exist but the signal does not. `job_schedule_state` stores `interval_seconds` and `paused` per provider per job type, so a per-provider rate multiplier is a more natural hook than a skip list. What is missing is a persisted health signal to drive it: `storage_provider` carries `isActive` / `isApproved` (both refreshed from the on-chain registry, neither derived from our check outcomes) and no failure counters. Failure history only exists as individual `deal` / `retrieval` rows, with nothing aggregating it back onto the provider. That aggregate is probably the first piece of work regardless of which throttling policy we land on, and it is worth checking against #280, which needs a similar signal.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with apps/backend/src/common/sp-blocklist.ts and apps/backend/src/jobs/jobs.service.ts, especially ensureScheduleRows(), then inspect job_schedule_state and storage_provider as described. Review the approval methodology and compare the proposal with #280. Done means a written brownout design answers the five open questions, records a go/no-go decision, and files implementation issues if approved.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- backend, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100