Duplicate blocks in cache silently block subgraph indexing

Abierto
#6,382 1 comentario 0 reacciones 1 asignado Ver en GitHub

@incrypto32 ya está trabajando en esto.

Desde el 18/5/2026.

Evaluación

Este issue todavía no se ha evaluado.

Descripción

Summary

Duplicate blocks accumulating in the block cache (chainN.blocks) can cause subgraphs to stop indexing. The subgraph remains health=healthy and active=true but processes 0 blocks/s. The only symptom is an increasing blocks_behind count.

The error is only visible at DEBUG level:

Block stream produced a non-fatal error, error: block stream error Block 14500860 does not contain hash

This repeats every ~7 minutes indefinitely. The subgraph never fails or enters an unhealthy state — it just stops advancing.

Root cause

The block ingestor stores both branches of a reorg in the block cache without cleaning up orphaned entries. Under certain conditions (not fully characterized — it does not happen at every duplicate block), a subgraph hits a block where the hash it expects no longer resolves, causing the "does not contain hash" error loop.

It is unclear what exact combination of factors triggers the issue — the presence of duplicate blocks is a necessary condition, but there may be additional factors (e.g., specific reorg depth, timing of block ingestion vs subgraph processing).

Impact observed (production, ~250 subgraphs)

Chain Duplicate blocks Range checked Ratio
moonbeam 2,400+ 16,000 blocks ~15%
mainnet (Ethereum PoS) 1,382 full cache —
  • Blocked subgraphs can show very high reorg_count (779,742 for the moonbeam case)
  • The issue is silent — no WARN/ERROR at default log level, health stays healthy
  • Observed on chains using RPC polling (not firehose)

Workaround

Remove duplicate blocks from the cache and perform a short rewind:

-- Find duplicate blocks
SELECT number, count(*) FROM chainN.blocks
GROUP BY number HAVING count(*) > 1;

# Remove duplicates (or truncate the whole chain cache)
graphman chain check-blocks <chain> by-range -f <from> -t <to> --delete-duplicates
# or: graphman chain truncate <chain>

# Then rewind the stuck subgraph with the correct block hash
graphman rewind --force --block-number <N> --block-hash <correct_hash> <deployment>

Related

  • #4792 — reports the same duplicate block storage issue

Environment

  • graph-node v0.41.1
Lenguaje dominante
Rust
Estrellas
3.2k
Forks
1.1k
Merge medio
4 d 1 h
PR fusionados (30 d)
1

Guía de contribución

Abrir la guía de contribución

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de graphprotocol/graph-node

Todos los issues de graphprotocol/graph-node

Issues similares

Más issues de Rust

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.