Calibration eRPC 429 storms: deal flow resolveByProviderId fans out O(active data sets) eth_calls per deal
- Dominant language
- TypeScript
- Stars
- 3
- Forks
- 9
- Avg merge
- 4d 14h
- Merged PRs (30d)
- 6
Description
## Problem
dealbot deal jobs call `synapse.storage.createContext({ providerId })` → synapse-sdk `StorageContext.resolveByProviderId`, which fires `getActivePieceCount` + `getAllDataSetMetadata` **per active data set** (batch size ≥50) to metadata-select a set to reuse. On calibration dealbot has up to ~196 active data sets with a single provider (see #605), so one resolve fires ~132 `eth_call`s, ×2–3 per deal, with all SPs phase-locked to the same scheduler tick → **300–500 `eth_call`/s bursts** at the top of the minute. These drain chain.love's calib token (currently 1000 req/min) in ~2–3s → eRPC `ErrUpstreamsExhausted` 429 storms (~9,748 chain.love 429s / 20h, ~12k Ankr-free budget trips). Mainnet is clean today (lower volume) but runs the same armed code path.
## Evidence
- Decoded burst selectors (exhausted eth_calls, 6h): `getActivePieceCount` 2,219, `getAllDataSetMetadata` 656 — standalone, no Multicall3 (matches `resolveByProviderId`, not the piece-cleanup `aggregate3` path).
- Per-second 429s: up to 337 in a single second; job-window correlation deal-flow 217 vs piece-cleanup 2.
- `getActivePieceCount` is O(n)-in-pieces and OOG-reverts on large sets (FilOzone/pdp#265, FilOzone/synapse-sdk#851), so it dominates the failure sample.
## Fix plan
- **synapse-sdk (root cause):** tracked in FilOzone/synapse-sdk#631 — metadata-filter first, replace O(n) `getActivePieceCount > 0` with `getActivePiecesByCursor(limit=1)` (#848), bounded concurrency with real early-exit, `Promise.allSettled`. Validate with the rpc-tracer against a ~196-data-set fixture, then `patch-package` into dealbot.
- **infra eRPC guardrail:** disable `rateLimitAutoTune`, set static budget ≈ chain.love cap, exclude 429 from `failsafe.retry`.
- **dealbot hygiene:** bound active data-set accumulation (#605).
## Related
FilOzone/synapse-sdk#631, #851, #848; FilOzone/pdp#265; #603 (RPC timeout).
## Done criteria
- Calibration deal-flow `eth_call`s per deal bounded (not O(active data sets)).
- Calibration `ErrUpstreamsExhausted`/429 storms eliminated; mainnet stays clean after v1.9.0.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.