graphprotocol / graphprotocol/graph-node

Rules-based system to control index creation and similar attributes

Open
#4,325 6 comments 3 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Stale
Dominant language
Rust
Stars
3.2k
Forks
1.1k
Avg merge
4d 1h
Merged PRs (30d)
1

Description

There are a few things that can be set manually through graphman or directly by manipulating the database that affect how subgraph data is stored, and can help query performance. Those items include:

  • custom indexes (multicolumn indexes or partial indexes)
  • creation of default attribute indexes (a lot of them are never used)
  • account-like optimization
  • statistics target
  • amount of history

An example of what that could like like in YAML form is

# Sketch of what an index definition file could look like. The file will
# contain many entries like this one
- rule: Messari analytics subgraphs
  match:
    # All subgraphs whose name matches this regex will be treated according
    # to these actions
    name: messari/.*
  actions:
    # Keep 10,000 blocks of history
    - history: 10000   
    - entity:
        name: LiquidityPoolSnapshot
        index:
          - on: [pool, timestamp]
    - entity:
        name: LiquidityPool
        # Turn off attribute indexes
        default_indexes: false
        # Set the table to account-like
        account-like: true
        # Increase the statistics target
        stats-target: 300
        # Create some custom indexes
        index:
          # Use a GiST index since we use block_range
          - on: [block_range, timestamp, id]
            method: gist
          # A partial index that only covers blocks after a certain point
          # TODO: since the block number means different things on
          # different networks, we should either use a timespan or let users
          # match on network
          - on: [timestamp, id]
            after: 15664000
          - on: [pool_address, block_range, timestamp]
            method: gist

We need to think more about the way in which operators interact with these rules. Some requirements:

  • changing rules should not require a restart of graph-node
  • it should be possible to list rules, delete them etc.
  • rules should be applied when a subgraph is deployed, but there should also be tooling to apply a new rule to an existing deployment.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Use graphman and the existing database-manipulation paths as entry points; first map how current index, history, and attribute settings are represented and applied. Define how rules are listed, deleted, reloaded without restarting graph-node, applied at deployment, and applied to existing deployments; done means these workflows are specified and covered by tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.