graphprotocol / graphprotocol/graph-node

[Bug] blockHashFromNumber returns null after 10s when indexing permits are exhausted (0.43.0+)

オープン
#6,720 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る

まだ誰も着手していません。

主要言語
Rust
スター
3.2k
フォーク
1.1k
平均マージ
4日 1時間
マージ済み PR(30日)
1

説明

Bug report

Since #6491 (0.43.0), Chain::block_pointer_from_number checks chain_store.block_ptrs_by_numbers before calling the RPC. If the block isn't in recent_blocks_cache, that lookup goes to blocks_from_store_by_numbers, which calls pool.get_permitted(). So it waits for the indexing semaphore (pool_size + GRAPH_EXTRA_QUERY_PERMITS), with no timeout.

The index-node resolver wraps the lookup in BLOCK_HASH_FROM_NUMBER_TIMEOUT (10s). When indexing holds every permit, blockHashFromNumber returns null and the RPC is never tried. 0.41.1 went straight to the RPC.

indexer-agent reports this as Failed to query graph node for blockHashFromNumber: no data returned (IE070), which blocks epoch resolution and POI closes. Epoch start blocks are hours old, so they always take the store path.

Repro: v0.45.0 against 0.41.1, pool_size = 2, mainnet and gnosis ingestors. To exhaust the permits I held LOCK TABLE public.ethereum_networks IN EXCLUSIVE MODE. Both ingestors then wait in the chain head UPDATE while holding their permit, which I confirmed in pg_stat_activity. It's a stand-in for write load.

build block 25961537 (not in recent cache) recent block
0.41.1 hash, 0.08s hash, 0.12s
0.45.0 null, 10.04s hash, 0.02s
0.45.0 with the cache check removed hash, 0.08s hash, 0.08s

store_semaphore_wait_ms doesn't show it. The wait is recorded only once the permit is acquired, and this lookup is cancelled before that (the metric read 52 during the null).

GRAPH_EXTRA_QUERY_PERMITS=4 does return the hash, in 5.11s, because the lookup times out on the pool connection and falls through to the RPC. That timeout calls state_tracker.mark_unavailable though, so it isn't a good workaround.

Suggested fix: drop the cache check from block_pointer_from_number, as #6537 did for is_on_main_chain, because the block cache isn't finality-aware here either. The callers are the index-node resolver, the registrar on deploy/graft and the public POI fallback, none of them hot. Still present on master (6838f4e3c).

Can open a PR with that if the direction sounds right?

Relevant log output
WARN Failed to fetch block hash from number, error: deadline has elapsed, block_number: 25961537, chain: ethereum, network: mainnet, component: IndexNodeServer > IndexNodeResolver
INFO Query timing (GraphQL), block: 0, query_time_ms: 10002, variables: null, query: { version { version } blockHashFromNumber(network: "mainnet", blockNumber: 25961537) } , subgraph_id: indexnode, component: IndexNodeServer
IPFS hash

No response

Subgraph name or link to explorer

No response

Some information to help us out
  • Tick this box if this bug is caused by a regression found in the latest release.
  • Tick this box if this bug is specific to the hosted service.
  • I have searched the issue tracker to make sure this issue is not a duplicate.
OS information

macOS

コントリビューションガイド

コントリビューションガイドを開く

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

調査の方向性

Chain::block_pointer_from_number から開始し、chain_store.block_ptrs_by_numbers をたどって blocks_from_store_by_numbers まで進み、その後インデックスノードリゾルバの呼び出し元を調べます。インデックス作成用の permit が使い果たされた状態で再現し、blockHashFromNumber が RPC に到達して既存のタイムアウト内にハッシュを返すこと、また状態を利用不可としてマークしないことを確認します。関連するテストが存在する場合は、回帰テストのカバレッジを追加します。

索引モデルが issue の本文から書いたものです。

評価

技術スタック
postgresql, rust
領域
backend, blockchain, databases
issue の種類
バグ
難易度
3/5
見積もり時間
1〜2日
活発さ
活発
明瞭さ
明確に書かれている
初心者へのやさしさ
74/100

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。