graphprotocol / graphprotocol/indexer

indexer-agent v0.25.10: TAPv2 GraphTallyCollector runs one cycle after boot, then silently never again

Open
#1,254 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
TypeScript
Stars
262
Forks
148
PR merge metrics
No merged PRs in 30d

Description

Summary

indexer-agent v0.25.10's TAPv2 collector (GraphTallyCollector) runs exactly one redemption cycle after process start and then never again, while the rest of the agent (reconcile loops, NetworkMonitor, actions queue) keeps running normally. The process looks fully healthy — pod Ready, thousands of log lines per 10 minutes — but no [TAPv2] log line appears for 6+ hours after the initial cycle. We hit this twice in three days, and it delayed redemption of ~42k GRT until a manual restart.

Environment

  • ghcr.io/graphprotocol/indexer-agent:v0.25.10
  • Arbitrum One, Horizon enabled (INDEXER_HORIZON__ENABLED=true on the paired indexer-rs stack)
  • Indexer 0xEdca8740873152fF30a2696Add66D1ab41882beB (also observed cycle-once behaviour on a second identity, 0x3717CE…, same version)

Observed timeline (occurrence 2, 2026-08-30 UTC)

  1. rollout restart → agent boots, and ~2 minutes later runs a full TAPv2 cycle: Query RAVs eligible for redemptionRAVs passed escrow balance checkSubmit RAVs on chain via batched multicall → (at that time each collect reverted for an unrelated on-chain reason).
  2. For the following 6 hours: zero [TAPv2] lines. Meanwhile the same window shows ~1,400 log lines/10 min of normal Agent/NetworkMonitor activity. 323 RAVs remained eligible the whole time (above threshold, allocations closed).
  3. Second rollout restart → TAPv2 cycle fires within 2 minutes → Batch redemption completed, 323 RAVs redeemed (~42k GRT).

Occurrence 1 (2026-08-28) was the harder variant: the agent booted while its Postgres was briefly unreachable, logged Unhandled promise rejection at startup, and then produced no log lines at all for 2 days while the pod stayed Ready (liveness only checks the HTTP port).

Expected

The RAV redemption loop runs on its interval for the lifetime of the process, or the process exits/fails visibly when the loop dies.

Suspicion

The one-cycle-then-silence pattern suggests the collector's loop promise rejects/errors after the first iteration (e.g., when a batch fails validation) without being rescheduled or surfacing — consistent with the startup Unhandled promise rejection variant being fatal to individual loops but not the process.

Ask

  • Reschedule/restart the TAPv2 loop on error, or crash the process so orchestration restarts it.
  • A heartbeat log line (or metric) for the collector loop so operators can alert on its absence.

Happy to provide fuller logs privately.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the TAPv2 GraphTallyCollector entry point and trace the redemption loop promise through its first cycle and error paths. Reproduce the one-cycle-then-silence behavior using the reported logs and verify that the loop either continues for the process lifetime or that its failure becomes visible and causes a restart.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
backend, observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.