[Bug] Rewinding a subgraph causes a constraint violation in graph-node that in turn causes indexer-agent to crashloop

未关闭
#5,316 3 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

还没有人认领这个 Issue。

评估

难度
4/5
预计耗时
3-5 天
新手友好度
35/100
Issue 类型
缺陷
描述清晰度
需要澄清
活跃度
停滞
技术栈
postgresql, rust

调研方向

首先检查 graph-node 和 indexer-agent 的日志输出,尤其是 packages/indexer-common/dist/graph-node.js 和 packages/indexer-common/dist/errors.js,然后调查 entityCount 约束相关的 rewind 和索引状态路径。重现 graphman 的 rewind 或 drop 场景,并确认 graph-node 恢复同步且 indexer-agent 不再进入 crashloop。

由索引模型根据 Issue 内容生成。

描述

bug Stale
Bug report

graph-node:v0.34.1
indexer-agent:v0.20.22

Activities that were undertaken before observing this bug:

  1. Cleared call_cache for Arbitrum as part of a complex subgraph sync performance troubleshooting exercise via psql
  2. Rewound a specific problematic subgraph, Silo Finance Arbitrum, QmTMKqty5yZvZtB3SwzXUG92aZUH1YQw3VjByGw4wgaMhW to block 1 using graphman
  3. Observed the above subgraph syncing to ~130m blocks, then stalled.
  4. Checked graph-node logs and found related error (see log output)
  5. Observed indexer-agent complaining about same issue and crash looping - cannot use the agent at all right now to manage subgraphs (see log output)

IMPACT: Production Indexer at risk; we cannot manage our online and offline allocations while we have this issue - ideally need a temp fix for the specific symptoms. Would graphman drop resolve the issue? Would the graph-node and indexer-agent be able to handle that and start syncing the sub again from scratch given this is a subgraph in flight with live allocations?

Relevant log output
----- GRAPH-NODE
Apr 04 13:24:34.037 ERRO Subgraph instance failed to run: internal constraint violated: Subgraph writer for QmTMKqty5yZvZtB3SwzXUG92aZUH1YQw3VjByGw4wgaMhW[sgd622] is not running, sgd: 622, subgraph_id: QmTMKqty5yZvZtB3SwzXUG92aZUH1YQw3VjByGw4wgaMhW, component: SubgraphInstanceManager
Apr 04 13:48:18.741 WARN Price provider Removed: 0x8dca64a43865454f41aa1a3cf0140eb89f2c08aa53871235ecbe46b6a309a1e3, data_source: PriceProvidersRepository, sgd: 622, subgraph_id: QmTMKqty5yZvZtB3SwzXUG92aZUH1YQw3VjByGw4wgaMhW, component: SubgraphInstanceManager > UserMapping
Apr 04 13:48:18.742 ERRO Oracle was not found when trying to remove it at txn: 0x8dca64a43865454f41aa1a3cf0140eb89f2c08aa53871235ecbe46b6a309a1e3, data_source: PriceProvidersRepository, sgd: 622, subgraph_id: QmTMKqty5yZvZtB3SwzXUG92aZUH1YQw3VjByGw4wgaMhW, component: SubgraphInstanceManager > UserMapping

----- INDEXER-AGENT
{"level":50,"time":1712241706593,"pid":1,"hostname":"268ad9e1400b","name":"IndexerAgent","component":"GraphNode","err":{"type":"IndexerError","message":"Failed to query indexing status API","stack":"IndexerError: Failed to query indexing status API\n    at indexerError (/opt/indexer/packages/indexer-common/dist/errors.js:173:12)\n    at GraphNode.<anonymous> (/opt/indexer/packages/indexer-common/dist/graph-node.js:146:55)\n    at Generator.next (<anonymous>)\n    at fulfilled (/opt/indexer/packages/indexer-common/dist/graph-node.js:5:58)\n    at processTicksAndRejections (node:internal/process/task_queues:96:5)","code":"IE018","explanation":"https://github.com/graphprotocol/indexer/blob/main/docs/errors.md#ie018","cause":{"type":"CombinedError","message":"[GraphQL] Store error: internal constraint violated: the entityCount for QmTMKqty5yZvZtB3SwzXUG92aZUH1YQw3VjByGw4wgaMhW is not representable as a u64","name":"CombinedError","graphQLErrors":[{"message":"Store error: internal constraint violated: the entityCount for QmTMKqty5yZvZtB3SwzXUG92aZUH1YQw3VjByGw4wgaMhW is not representable as a u64"}],"response":{"size":0,"timeout":0}}},"msg":"Failed to query indexing status API"}
IPFS hash

No response

Subgraph name or link to explorer

https://thegraph.com/explorer/subgraphs/2ufoztRpybsgogPVW6j9NTn1JmBWFYPKbP7pAabizADU?view=Overview&chain=arbitrum-one

Some information to help us out
  • Tick this box if this bug is caused by a regression found in the latest release.
  • Tick this box if this bug is specific to the hosted service.
  • I have searched the issue tracker to make sure this issue is not a duplicate.
OS information

Linux

主要语言
Rust
星标
3.2k
派生
1.1k
平均合并
4 天 1 小时
30 天内合并 PR
1

贡献指南

打开贡献指南

从这里开始

  1. 先读完整个 Issue,再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库,在一个分支上完成修改。
  4. 提交 Pull Request,并在描述里引用这个 Issue 编号。

graphprotocol/graph-node 的其他 Issue

查看 graphprotocol/graph-node 的全部 Issue

相似的 Issue

更多 Rust Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。