graphprotocol / graphprotocol/graph-node
[Bug] blockHashFromNumber returns null after 10s when indexing permits are exhausted (0.43.0+)
Nessuno ha ancora preso questa issue.
- Lingua principale
- Rust
- Stelle
- 3.2k
- Fork
- 1.1k
- Merge medio
- 4g 1h
- PR unite (30g)
- 1
Descrizione
Bug report
Since #6491 (0.43.0), Chain::block_pointer_from_number checks chain_store.block_ptrs_by_numbers before calling the RPC. If the block isn't in recent_blocks_cache, that lookup goes to blocks_from_store_by_numbers, which calls pool.get_permitted(). So it waits for the indexing semaphore (pool_size + GRAPH_EXTRA_QUERY_PERMITS), with no timeout.
The index-node resolver wraps the lookup in BLOCK_HASH_FROM_NUMBER_TIMEOUT (10s). When indexing holds every permit, blockHashFromNumber returns null and the RPC is never tried. 0.41.1 went straight to the RPC.
indexer-agent reports this as Failed to query graph node for blockHashFromNumber: no data returned (IE070), which blocks epoch resolution and POI closes. Epoch start blocks are hours old, so they always take the store path.
Repro: v0.45.0 against 0.41.1, pool_size = 2, mainnet and gnosis ingestors. To exhaust the permits I held LOCK TABLE public.ethereum_networks IN EXCLUSIVE MODE. Both ingestors then wait in the chain head UPDATE while holding their permit, which I confirmed in pg_stat_activity. It's a stand-in for write load.
| build | block 25961537 (not in recent cache) | recent block |
|---|---|---|
| 0.41.1 | hash, 0.08s | hash, 0.12s |
| 0.45.0 | null, 10.04s |
hash, 0.02s |
| 0.45.0 with the cache check removed | hash, 0.08s | hash, 0.08s |
store_semaphore_wait_ms doesn't show it. The wait is recorded only once the permit is acquired, and this lookup is cancelled before that (the metric read 52 during the null).
GRAPH_EXTRA_QUERY_PERMITS=4 does return the hash, in 5.11s, because the lookup times out on the pool connection and falls through to the RPC. That timeout calls state_tracker.mark_unavailable though, so it isn't a good workaround.
Suggested fix: drop the cache check from block_pointer_from_number, as #6537 did for is_on_main_chain, because the block cache isn't finality-aware here either. The callers are the index-node resolver, the registrar on deploy/graft and the public POI fallback, none of them hot. Still present on master (6838f4e3c).
Can open a PR with that if the direction sounds right?
Relevant log output
WARN Failed to fetch block hash from number, error: deadline has elapsed, block_number: 25961537, chain: ethereum, network: mainnet, component: IndexNodeServer > IndexNodeResolver
INFO Query timing (GraphQL), block: 0, query_time_ms: 10002, variables: null, query: { version { version } blockHashFromNumber(network: "mainnet", blockNumber: 25961537) } , subgraph_id: indexnode, component: IndexNodeServer
IPFS hash
No response
Subgraph name or link to explorer
No response
Some information to help us out
- Tick this box if this bug is caused by a regression found in the latest release.
- Tick this box if this bug is specific to the hosted service.
- I have searched the issue tracker to make sure this issue is not a duplicate.
OS information
macOS
Guida per i contributori
Apri la guida per i contributori
Come iniziare
- Leggi tutta la issue e poi la guida ai contributi del progetto.
- Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
- Fai un fork del repository e lavora su un branch.
- Apri una pull request che faccia riferimento al numero della issue.
Direzione di ricerca
Inizia da Chain::block_pointer_from_number e segui chain_store.block_ptrs_by_numbers fino a blocks_from_store_by_numbers, quindi esamina i chiamanti del resolver dei nodi di indice. Riproduci il problema con i permessi di indicizzazione esauriti e verifica che blockHashFromNumber raggiunga l’RPC e restituisca l’hash entro il timeout esistente, senza contrassegnare lo stato come non disponibile; aggiungi una copertura di regressione se esiste un test pertinente.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Valutazione
- Stack tecnologico
- postgresql, rust
- Ambito
- backend, blockchain, databases
- Tipo di issue
- Bug
- Difficoltà
- 3/5
- Tempo stimato
- 1-2 giorni
- Stato di attività
- Attiva
- Chiarezza
- Specificata chiaramente
- Idoneità per principianti
- 74/100