graphprotocol / graphprotocol/graph-node

[Bug] blockHashFromNumber returns null after 10s when indexing permits are exhausted (0.43.0+)

Aperta
#6,720 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

Nessuno ha ancora preso questa issue.

Lingua principale
Rust
Stelle
3.2k
Fork
1.1k
Merge medio
4g 1h
PR unite (30g)
1

Descrizione

Bug report

Since #6491 (0.43.0), Chain::block_pointer_from_number checks chain_store.block_ptrs_by_numbers before calling the RPC. If the block isn't in recent_blocks_cache, that lookup goes to blocks_from_store_by_numbers, which calls pool.get_permitted(). So it waits for the indexing semaphore (pool_size + GRAPH_EXTRA_QUERY_PERMITS), with no timeout.

The index-node resolver wraps the lookup in BLOCK_HASH_FROM_NUMBER_TIMEOUT (10s). When indexing holds every permit, blockHashFromNumber returns null and the RPC is never tried. 0.41.1 went straight to the RPC.

indexer-agent reports this as Failed to query graph node for blockHashFromNumber: no data returned (IE070), which blocks epoch resolution and POI closes. Epoch start blocks are hours old, so they always take the store path.

Repro: v0.45.0 against 0.41.1, pool_size = 2, mainnet and gnosis ingestors. To exhaust the permits I held LOCK TABLE public.ethereum_networks IN EXCLUSIVE MODE. Both ingestors then wait in the chain head UPDATE while holding their permit, which I confirmed in pg_stat_activity. It's a stand-in for write load.

build block 25961537 (not in recent cache) recent block
0.41.1 hash, 0.08s hash, 0.12s
0.45.0 null, 10.04s hash, 0.02s
0.45.0 with the cache check removed hash, 0.08s hash, 0.08s

store_semaphore_wait_ms doesn't show it. The wait is recorded only once the permit is acquired, and this lookup is cancelled before that (the metric read 52 during the null).

GRAPH_EXTRA_QUERY_PERMITS=4 does return the hash, in 5.11s, because the lookup times out on the pool connection and falls through to the RPC. That timeout calls state_tracker.mark_unavailable though, so it isn't a good workaround.

Suggested fix: drop the cache check from block_pointer_from_number, as #6537 did for is_on_main_chain, because the block cache isn't finality-aware here either. The callers are the index-node resolver, the registrar on deploy/graft and the public POI fallback, none of them hot. Still present on master (6838f4e3c).

Can open a PR with that if the direction sounds right?

Relevant log output
WARN Failed to fetch block hash from number, error: deadline has elapsed, block_number: 25961537, chain: ethereum, network: mainnet, component: IndexNodeServer > IndexNodeResolver
INFO Query timing (GraphQL), block: 0, query_time_ms: 10002, variables: null, query: { version { version } blockHashFromNumber(network: "mainnet", blockNumber: 25961537) } , subgraph_id: indexnode, component: IndexNodeServer
IPFS hash

No response

Subgraph name or link to explorer

No response

Some information to help us out
  • Tick this box if this bug is caused by a regression found in the latest release.
  • Tick this box if this bug is specific to the hosted service.
  • I have searched the issue tracker to make sure this issue is not a duplicate.
OS information

macOS

Guida per i contributori

Apri la guida per i contributori

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Direzione di ricerca

Inizia da Chain::block_pointer_from_number e segui chain_store.block_ptrs_by_numbers fino a blocks_from_store_by_numbers, quindi esamina i chiamanti del resolver dei nodi di indice. Riproduci il problema con i permessi di indicizzazione esauriti e verifica che blockHashFromNumber raggiunga l’RPC e restituisca l’hash entro il timeout esistente, senza contrassegnare lo stato come non disponibile; aggiungi una copertura di regressione se esiste un test pertinente.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
postgresql, rust
Ambito
backend, blockchain, databases
Tipo di issue
Bug
Difficoltà
3/5
Tempo stimato
1-2 giorni
Stato di attività
Attiva
Chiarezza
Specificata chiaramente
Idoneità per principianti
74/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.