graphprotocol / graphprotocol/graph-node

[Bug] Rollback subgraph automatically if it lands on a fork when connected to Firehose

Open
#5,156 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug firehose Stale
Dominant language
Rust
Stars
3.2k
Forks
1.1k
Avg merge
4d 1h
Merged PRs (30d)
1

Description

Bug report

The Firehose fork database is kept for a limited amount of time. Graph-node could be more intelligent and do that automatically when the fork can't be resolved.

Forks also don't resolve if you configure multiple firehose providers and one sees the fork and the other does not.

Graph Node needs to throw away its cursor in this case and reverts back to known-final, but we need guidance on how to reliably detect the error message. Having a specific error handling process would be a key requirement (without parsing text messages).

Relevant log output
Jan 22 04:15:32.485 ERRO Unable to connect to endpoint: status: Unknown, message: "transport error", details: [], metadata: MetadataMap { headers: {} }: transport error: operation was canceled: connection closed: connection closed, provider: firehose, deployment: QmSjgMiYaD3qPqhDBf3ody1wuwPRuD5iJjdxd3qTuxJ8Dj, sgd: 765, subgraph_id: QmSjgMiYaD3qPqhDBf3ody1wuwPRuD5iJjdxd3qTuxJ8Dj, component: FirehoseBlockStream
Jan 22 04:16:23.776 ERRO An error occurred while streaming blocks: status: InvalidArgument, message: "cannot resolve cursor: missing link between blocks 18962442 and #18962520 (48814e410c9313b9cb4562522cb18523c7840d7aba760498f50b61d4ccfcb655): no forked-block file found with ID ending with f50b61d4ccfcb655.", details: [], metadata: MetadataMap { headers: {} }, provider: firehose, deployment: QmSjgMiYaD3qPqhDBf3ody1wuwPRuD5iJjdxd3qTuxJ8Dj, sgd: 765, subgraph_id: QmSjgMiYaD3qPqhDBf3ody1wuwPRuD5iJjdxd3qTuxJ8Dj, component: FirehoseBlockStream
Jan 22 04:17:13.841 ERRO An error occurred while streaming blocks: status: InvalidArgument, message: "cannot resolve cursor: missing link between blocks 18962442 and #18962520 (48814e410c9313b9cb4562522cb18523c7840d7aba760498f50b61d4ccfcb655): no forked-block file found with ID ending with f50b61d4ccfcb655.", details: [], metadata: MetadataMap { headers: {} }, provider: firehose, deployment: QmSjgMiYaD3qPqhDBf3ody1wuwPRuD5iJjdxd3qTuxJ8Dj, sgd: 765, subgraph_id: QmSjgMiYaD3qPqhDBf3ody1wuwPRuD5iJjdxd3qTuxJ8Dj, component: FirehoseBlockStream
IPFS hash

No response

Subgraph name or link to explorer

No response

Some information to help us out
  • Tick this box if this bug is caused by a regression found in the latest release.
  • Tick this box if this bug is specific to the hosted service.
  • I have searched the issue tracker to make sure this issue is not a duplicate.
OS information

Linux

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the FirehoseBlockStream path named in the logs and how it handles the reported “cannot resolve cursor” error. Determine how fork loss and conflicting Firehose providers can be detected without parsing text messages. Done means the subgraph can discard its cursor and revert to known-final reliably, with coverage for the reported failure modes.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend, blockchain, distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.