Non-deterministic eth_call reads silently corrupt indexing (POI divergence)
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 5
- Forks
- 7
- Avg merge
- 4h 1m
- Merged PRs (30d)
- 6
Description
## Summary
Several handlers read contract state with `eth_call`. graph-node runs each call against the RPC at the historical block and trusts the response, so a degraded or lagging backend yields a plausible-but-wrong value that gets persisted. `try_` never aborts at the call site — it takes the `reverted` branch and writes a fallback — so the damage surfaces later as wrong data, as an abort in an unrelated handler, or as POI divergence between indexers running identical code.
We hit the delayed-abort variant at block **144056642** and shipped a targeted patch (#249). This issue tracks the general class.
**Rule of thumb:** a `try_` fallback is safe only when the revert branch is bounded to a known block range or network **and** its value is indistinguishable in effect from the truth. Otherwise it is silently manufacturing state.
## Call sites
| Call site | Fallback | Blast radius | Aborts | Self-heals |
|---|---|---|---|---|
| `try_getFirstTranscoderInPool` `roundsManager.ts:54` | `EMPTY_ADDRESS` → loop skipped | no `Pool` for the whole round | was yes, fixed by #249 | no |
| `try_getTotalBonded` `roundsManager.ts:66` | `0` | `round`/`protocol`/`day` stake + 5 fields skipped | no | partially |
| `try_slot0` `helpers.ts:472` | price `0` | 7 cumulative USD accumulators | no | **no** |
| `try_getGlobalTotalSupply` `livepeerToken.ts:23,76` | `totalSupply ± amount` | `totalSupply`, `participationRate` | no | yes |
| `getNextTranscoderInPool` (bare) `roundsManager.ts:130` | n/a — reverts abort | truncated walk on a wrong-but-valid answer | yes, loudly | no |
**1 — `getFirstTranscoderInPool`.** The original incident. A revert leaves `currentTranscoder` at `EMPTY_ADDRESS`, the loop at `roundsManager.ts:122` never runs, and no `Pool` exists for the round. A revert means "the pool is empty", impossible post-launch, so the fallback is never legitimately taken.
**2 — `getTotalBonded`.** Same handler, same block, same RPC as #1 — they fail together. Still unguarded. Zero is written to `round`/`protocol`/`day` (`roundsManager.ts:75,226,229`), and the `gt(ZERO_BD)` guard at `roundsManager.ts:232` then skips `participationRate`, `delegatorsCount`, `inflation`, `numActiveTranscoders`, `activeTranscoderCount` on both `round` and `day`. `protocol.totalActiveStake = 0` persists until the next `newRound`, zeroing `day.totalActiveStake` on every `mint`/`burn`/`winningTicketRedeemed` in between.
**3 — `slot0`.** Price `0` flows via `getEthPriceUsd()` into seven `.plus()` accumulators (`ticketBroker.ts:81,90,105,109,130,138,151`), so it permanently under-counts USD volume. `getPriceForPair` also returns `ZERO_BD` by design off-Arbitrum, making `0` an overloaded sentinel — "wrong network", "pool not yet deployed", and "RPC hiccup" are indistinguishable. The LPT/ETH pool `0x4fd47e…` has no code at the arbitrum-one start blocks, so the zero path is genuinely exercised early in a sync.
**4 — `getGlobalTotalSupply`.** Written correctly, listed for completeness. The fallback `protocol.totalSupply ± amount` is arithmetically right in normal operation and resnaps to chain truth on the next successful call. Its legit-revert window is real and bounded (`MinterV1` covers arbitrum-one `5856338 → 6253359`). No action proposed.
**5 — bare calls.** Aborting on revert is the *preferable* behavior for a determinism hazard. But a wrong-but-valid response still corrupts silently: a zero address from `getNextTranscoderInPool` exits the walk mid-list and every transcoder after it gets no `Pool`. Only event-derived membership fixes that.
## Incident (instance of case 1)
Block **144056642**, round 3154, transcoder `0xBAc7744ADA4AB1957CBaAFEF698B3c068bEB4fe0`, tx `0xd308120046d57aac4a66d1466d68b602cb754e23a7c1d2dcbcadca8f91469081`. Studio halted on `pool!.rewardTokens` with a null `Pool`. Network indexer `QmUW5c…` passed the same block on the same code, which is what identified it as indexer-side non-determinism. `scripts/check-reward-pool.cjs` confirms 99 transcoders in the on-chain pool at that block with the transcoder present, so a correct `newRound()` would have built the `Pool`.
## Patched
**#249 (merged)** added `createOrLoadPool()` and routed `newRound`, `reward`, and `ticketBroker` through it, so a missing `Pool` is built on demand instead of dereferenced with `!`. Shipped with #245 (nullable `Transaction.to`).
Fixes the abort. Does **not** fix: the truncated enumeration (lazily-created Pools carry a different snapshot, and transcoders that neither rewarded nor redeemed fees still get none), POI divergence, or cases 2 and 3.
#246 and #247 are superseded by #245 and #249 and can be closed.
## Next
**Structural** — derive pool membership from events (activation/deactivation/reward) instead of state reads, removing the enumeration entirely.
**Hardening**
- `getTotalBonded`: carry forward the last known `protocol.totalActiveStake` instead of `zero()`, mirroring the sane-fallback pattern already used for `getGlobalTotalSupply`.
- Raise `log.info` → `log.error` at `roundsManager.ts:56,68`. Both reverts are impossible post-launch, so `info` makes a determinism hazard invisible. `log.critical` would abort, which for POI divergence is arguably more honest — worth an explicit decision.
- Disambiguate the `getPriceForPair` zero sentinel.
- Use the existing `safeDiv` (`helpers.ts:448`) at `livepeerToken.ts:43,95`, where the guard checks the numerator but leaves `totalSupply` unguarded and `BigDecimal.div` by zero panics. Latent, not active — reaching it needs `totalSupply == 0`, unrealistic post-launch.
**Operational** — index against a consistent full-archive RPC, not a load-balanced pool that can hit a pruned or lagging backend.
## Diagnosis
- `scripts/check-reward-pool.cjs ` — confirms whether a transcoder was in the on-chain pool at a block, isolating indexer-side anomalies. `eth_call` only, works against any archive RPC.
- `docs/debugging.md` — reproducing an indexing crash locally against Arbitrum from a chosen `startBlock`.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the listed call sites in roundsManager.ts, helpers.ts, livepeerToken.ts, and ticketBroker.ts, then run scripts/check-reward-pool.cjs against a reward transaction. Read docs/debugging.md for local reproduction against Arbitrum. Done means the selected structural or hardening change prevents incorrect eth_call results from silently producing divergent indexed state.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- backend, data
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100