graphprotocol / graphprotocol/indexer

indexer-agent reconciliation loop overrides manual graphman pause

Open
#1,171 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
TypeScript
Stars
262
Forks
148
PR merge metrics
No merged PRs in 30d

Description

Summary

When using graphman pause <deployment> to manually pause a subgraph, the indexer-agent's reconcileDeployments loop automatically resumes it within minutes by calling subgraph_deploy on the graph-node JSON-RPC admin API. This makes it impossible to keep a subgraph paused for maintenance operations such as graphman rewind.

Steps to reproduce

  1. graphman pause <IPFS_HASH>
  2. Wait 2-5 minutes (one reconciliation cycle)
  3. graphman info <IPFS_HASH> --status → shows Paused: false

The subgraph has decisionBasis: always in indexer rules.

Expected behavior

The reconciliation loop should check the paused_at field before calling subgraph_deploy. If a deployment was explicitly paused via graphman, the agent should not resume it.

Impact

  • Cannot perform graphman rewind safely — the subgraph resumes indexing during the rewind, causing data inconsistencies
  • Any maintenance operation requiring a temporary stop is compromised
  • Operators must resort to workarounds like reassigning to a non-existent node (e.g. graphman reassign <hash> maintenance_node_0) to prevent the agent from resuming the subgraph

Root cause

In packages/indexer-agent/src/agent.ts, the reconcileDeployments function calculates target deployments based on indexer rules and calls this.graphNode.ensure() for each one. There is no check for whether a deployment is currently paused before calling ensure. If the deployment is in the target list (active allocation or decisionBasis: always/offchain), it will be re-deployed, which implicitly resumes it.

Suggested fix

Before calling ensure for a deployment, check if it is currently paused (paused_at IS NOT NULL). If paused, skip the ensure call. Optionally, add a --force flag to graphman pause or a new indexer rule field (e.g. maintenancePause: true) that the agent explicitly respects.

Related issues

  • #713 — Added --deployment-management manual mode as a workaround (disables entire reconciliation loop)
  • #853 — Aimed to make the agent respect the new pause mechanism, but the "use new pausing mechanism" task was never completed
  • graphprotocol/graph-node#4255 — Introduced first-class graphman pause/resume commands
  • graphprotocol/graph-node#5253 — graphman reassign should pause the subgraph first (still OPEN)
  • graphprotocol/graph-node#5110 — graphman rewind should use the pause mechanism (closed by stale bot, never implemented)

Current workaround

Reassign the subgraph to a non-existent node before maintenance:

graphman pause <IPFS_HASH>
graphman reassign <IPFS_HASH> maintenance_node_0
# perform maintenance (rewind, reindex, etc.)
graphman reassign <IPFS_HASH> index_node_0
graphman resume <IPFS_HASH>

The indexer-agent cannot resume the subgraph because maintenance_node_0 does not exist.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in packages/indexer-agent/src/agent.ts at reconcileDeployments and trace how target deployments reach this.graphNode.ensure(). Check how the paused_at field is exposed and reproduce the graphman pause scenario; done means an explicitly paused deployment is not resumed during reconciliation while normal deployments still reconcile.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.