graphprotocol / graphprotocol/graph-node

Spurious error on DB setup

Open
#3,369 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
3.2k
Forks
1.1k
Avg merge
4d 1h
Merged PRs (30d)
1

Description

DB setup has been observed to fail in the hosted service with the following error logs:

2022-03-16T19:29:16.264973019Z stderr F thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: JoinError::Panic(...)', /graph-node/store/postgres/src/connection_pool.rs:513:10
2022-03-16 19:29:16	
2022-03-16T19:29:16.264648288Z stderr F Mar 16 19:29:16.264 CRIT migrations failed, error: store error: failed to find plan for subquery ss, pool: main, shard: blocks_a, component: ConnectionPool
2022-03-16 19:29:16	
2022-03-16T19:29:16.264640948Z stderr F note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
2022-03-16 19:29:16	
2022-03-16T19:29:16.264553508Z stderr F thread 'tokio-runtime-worker' panicked at 'migrations failed: store error: failed to find plan for subquery ss', store/postgres/src/connection_pool.rs:987:13

This causes graph node to crash, until it the setup eventually succeeds for all shards. It's been observed on both PG 13 and PG 14 shards.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in store/postgres/src/connection_pool.rs at the reported failures around lines 513 and 987, and trace how migrations handle the failed subquery plan for shard blocks_a. Reproduce DB setup across PostgreSQL 13 and 14 shards, using the supplied logs and RUST_BACKTRACE=1 for more detail. Done means setup no longer crashes on this transient failure and completes successfully for all shards.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgresql, rust
Domain
backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.