graphprotocol / graphprotocol/indexer

[ISSUE] status endpoint crash

Open
#1,032 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
TypeScript
Stars
262
Forks
148
PR merge metrics
No merged PRs in 30d

Description

If I make a larger query like this:

graph indexer allocations get --deployment QmYNPk3Mabrx4VmxPKmBevdi11cfVUfTXVtgygG4M4m9tQ

where the SG allocation is very old or the query itself is very abstract and the API needs to perform a larger query, I notice that the status API endpoint is crashing with no logs related.

This command above produces this error:

✖ [GraphQL] Failed to query Epoch Block Oracle Subgraph

then I query the status network

graph indexer status network --network arbitrum-one

and I get this output with an error indicating that (possibly) the JVM crashed (again no logs and no pods restarted)

graph@indexer-cli:~$ graph indexer status network --network arbitrum-one
Registration
┌───────────────────────────────────────┬────────────────────────────────────────────┬─────────────────┬────────────┬─────────────────────────────────────┐
│ url                                   │ address                                    │ protocolNetwork │ registered │ location                            │
├───────────────────────────────────────┼────────────────────────────────────────────┼─────────────────┼────────────┼─────────────────────────────────────┤
│ https://indexer.upgrade.thegraph.com/ │ 0xbdfb5Ee5a2aBF4fc7bB1bD1221067aEf7f9dE491 │ eip155:42161    │ true       │ 89.99997854232788,36.57848596572876 │
└───────────────────────────────────────┴────────────────────────────────────────────┴─────────────────┴────────────┴─────────────────────────────────────┘

Endpoints
┌─────────┬─────────────────┬─────────────────────────────────────────────┬────────┐
│ name    │ protocolNetwork │ url                                         │ status │
├─────────┼─────────────────┼─────────────────────────────────────────────┼────────┤
│ service │ arbitrum-one    │ https://indexer.upgrade.thegraph.com/       │ down   │
├─────────┼─────────────────┼─────────────────────────────────────────────┼────────┤
│ status  │ arbitrum-one    │ https://indexer.upgrade.thegraph.com/status │ up     │
└─────────┴─────────────────┴─────────────────────────────────────────────┴────────┘
The following endpoint tests failed:

https://indexer.upgrade.thegraph.com/
  Test: http get https://indexer.upgrade.thegraph.com/
  Error: fetch failed
  Possible actions:
    - Make sure https://indexer.upgrade.thegraph.com/ can be resolved and reached from this machine
    - Make sure the port of https://indexer.upgrade.thegraph.com/ is set up correctly
    - Make sure the test command returns an HTTP status code < 400

Indexer Deployments
<--- Last few GCs --->

[26273:0x64f1df0]    83686 ms: Mark-Compact (reduce) 4090.9 (4116.2) -> 4090.9 (4109.2) MB, 35.44 / 0.00 ms  (+ 8.5 ms in 1 steps since start of marking, biggest step 8.5 ms, walltime since start of marking 48 ms) (average mu = 0.501, current mu = 0.311) [26273:0x64f1df0]    83747 ms: Mark-Compact (reduce) 4093.2 (4111.5) -> 4093.2 (4111.5) MB, 55.82 / 0.00 ms  (average mu = 0.340, current mu = 0.078) allocation failure; scavenge might not succeed


<--- JS stacktrace --->

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
----- Native stack trace -----

 1: 0xca5580 node::Abort() [node]
 2: 0xb781f9  [node]
 3: 0xeca4d0 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [node]
 4: 0xeca7b7 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [node]
 5: 0x10dc505  [node]
 6: 0x10f4388 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]
 7: 0x10ca4a1 v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
 8: 0x10cb635 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
 9: 0x10a8c86 v8::internal::Factory::NewFillerObject(int, v8::internal::AllocationAlignment, v8::internal::AllocationType, v8::internal::AllocationOrigin) [node]
10: 0x1503a16 v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [node]
11: 0x7daacfed9ef6
Aborted

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the failure with graph indexer allocations get --deployment QmYNPk3Mabrx4VmxPKmBevdi11cfVUfTXVtgygG4M4m9tQ, then query graph indexer status network --network arbitrum-one and inspect the status endpoint path involved. Trace the large query's memory use and add coverage if an existing test location is identified; done means the endpoint stays available without a Node.js heap-out-of-memory crash.

Written by the indexing model from the issue text.

Assessment

Tech stack
graphql, node.js, typescript
Domain
api, backend, cli
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.