graphprotocol / graphprotocol/graph-node

[Bug] v0.40.1 - "graphman reassign" operations fails ~ 50% of the time without any meaningful info in subgraph's logs

Abierto
#6,227 3 comentarios 3 reacciones 0 asignados Ver en GitHub

Nadie ha tomado este issue todavía.

bug Stale
Lenguaje dominante
Rust
Estrellas
3.2k
Forks
1.1k
Merge medio
4 d 1 h
PR fusionados (30 d)
1

Descripción

Bug report

Hi Team,
I'm using grapnode v0.40.1. Roughly 50% of my "graphman reassign" operations look like below:

  1. Subgraph is running perfectly fine (healthy, no lag, not paused) on indexer node X
  2. I invoke "graphman reassign SUBGRAPH_HASH indexer_node_Y" command and it finalises without problems
  3. I check the status of the subgraph with "graphman info --status" command after few minutes and the subgraph is stuck, it isn't processing any new blocks
  4. Target index node doesn't produce logs for this subgraph
  5. Prometheus metrics (for example "deployment_head") are available on the previous index node and on the new/target indexer node, on the previous index node they have misleading values as processing doesn't really take place on it anymore
  6. I have to perform at least 1 more reassign operation to fix the subgraphs, sometimes up to 5 reassignment operations have to be done
    I checked existing issues and it may be the case that https://github.com/graphprotocol/graph-node/issues/5253 is related so I've added a comment to it.

Can someone please take a look?

Relevant log output

IPFS hash

No response

Subgraph name or link to explorer

No response

Some information to help us out
  • Tick this box if this bug is caused by a regression found in the latest release.
  • Tick this box if this bug is specific to the hosted service.
  • I have searched the issue tracker to make sure this issue is not a duplicate.
OS information

None

Guía de contribución

Abrir la guía de contribución

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Línea de trabajo

Comienza reproduciendo el workflow intermitente de graphman reassign SUBGRAPH_HASH indexer_node_Y y, después, inspecciona graphman info --status, comparando los logs y deployment_head en ambos nodos indexer. Revisa el issue relacionado #5253 para obtener contexto. Se considera terminado cuando la reasignación comienza a procesarse de forma consistente en el nodo de destino, con métricas precisas y logs útiles cuando falla.

Escrito por el modelo de indexación a partir del texto del issue.

Evaluación

Stack tecnológico
rust
Área
cli, distributed-systems
Tipo de issue
Error
Dificultad
4/5
Tiempo estimado
3-5 días
Estado de actividad
Activo
Claridad
Bastante claro
Aptitud para principiantes
45/100

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.