graphprotocol / graphprotocol/graph-node

[Bug] blockHashFromNumber returns null after 10s when indexing permits are exhausted (0.43.0+)

Ouverte
#6,720 0 commentaires 0 réactions 0 personnes assignées Voir sur GitHub

Personne n'a encore pris cette issue.

Langage dominant
Rust
Étoiles
3.2k
Forks
1.1k
Merge moyen
4 j 1 h
PR mergées (30 j)
1

Description

Bug report

Since #6491 (0.43.0), Chain::block_pointer_from_number checks chain_store.block_ptrs_by_numbers before calling the RPC. If the block isn't in recent_blocks_cache, that lookup goes to blocks_from_store_by_numbers, which calls pool.get_permitted(). So it waits for the indexing semaphore (pool_size + GRAPH_EXTRA_QUERY_PERMITS), with no timeout.

The index-node resolver wraps the lookup in BLOCK_HASH_FROM_NUMBER_TIMEOUT (10s). When indexing holds every permit, blockHashFromNumber returns null and the RPC is never tried. 0.41.1 went straight to the RPC.

indexer-agent reports this as Failed to query graph node for blockHashFromNumber: no data returned (IE070), which blocks epoch resolution and POI closes. Epoch start blocks are hours old, so they always take the store path.

Repro: v0.45.0 against 0.41.1, pool_size = 2, mainnet and gnosis ingestors. To exhaust the permits I held LOCK TABLE public.ethereum_networks IN EXCLUSIVE MODE. Both ingestors then wait in the chain head UPDATE while holding their permit, which I confirmed in pg_stat_activity. It's a stand-in for write load.

build block 25961537 (not in recent cache) recent block
0.41.1 hash, 0.08s hash, 0.12s
0.45.0 null, 10.04s hash, 0.02s
0.45.0 with the cache check removed hash, 0.08s hash, 0.08s

store_semaphore_wait_ms doesn't show it. The wait is recorded only once the permit is acquired, and this lookup is cancelled before that (the metric read 52 during the null).

GRAPH_EXTRA_QUERY_PERMITS=4 does return the hash, in 5.11s, because the lookup times out on the pool connection and falls through to the RPC. That timeout calls state_tracker.mark_unavailable though, so it isn't a good workaround.

Suggested fix: drop the cache check from block_pointer_from_number, as #6537 did for is_on_main_chain, because the block cache isn't finality-aware here either. The callers are the index-node resolver, the registrar on deploy/graft and the public POI fallback, none of them hot. Still present on master (6838f4e3c).

Can open a PR with that if the direction sounds right?

Relevant log output
WARN Failed to fetch block hash from number, error: deadline has elapsed, block_number: 25961537, chain: ethereum, network: mainnet, component: IndexNodeServer > IndexNodeResolver
INFO Query timing (GraphQL), block: 0, query_time_ms: 10002, variables: null, query: { version { version } blockHashFromNumber(network: "mainnet", blockNumber: 25961537) } , subgraph_id: indexnode, component: IndexNodeServer
IPFS hash

No response

Subgraph name or link to explorer

No response

Some information to help us out
  • Tick this box if this bug is caused by a regression found in the latest release.
  • Tick this box if this bug is specific to the hosted service.
  • I have searched the issue tracker to make sure this issue is not a duplicate.
OS information

macOS

Guide de contribution

Ouvrir le guide de contribution

Par où commencer

  1. Lisez l'issue en entier, puis le guide de contribution du projet.
  2. Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
  3. Forkez le dépôt et travaillez sur une branche.
  4. Ouvrez une pull request qui référence le numéro de l'issue.

Piste de recherche

Commencez à Chain::block_pointer_from_number et suivez chain_store.block_ptrs_by_numbers jusqu’à blocks_from_store_by_numbers, puis examinez les appelants du résolveur de nœuds d’index. Reproduisez le problème avec les permis d’indexation épuisés et vérifiez que blockHashFromNumber atteint le RPC et renvoie le hash dans le délai d’attente existant, sans marquer l’état comme indisponible ; ajoutez une couverture de régression si un test pertinent existe.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Évaluation

Stack technique
postgresql, rust
Domaine
backend, blockchain, databases
Type d'issue
Bug
Difficulté
3/5
Temps estimé
1-2 jours
Activité
Active
Clarté
Clairement spécifiée
Accessibilité débutants
74/100

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.