graphprotocol / graphprotocol/graph-node

Feature: `.findMany` & `.saveMany`

Open
#3,618 4 comments 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Stale
Dominant language
Rust
Stars
3.2k
Forks
1.1k
Avg merge
4d 1h
Merged PRs (30d)
1

Description

Proposal

The injected entity API should also expose bulk operations like findMany, saveMany. This seems like a simple addition to me that would enable better bulk operations and indexing-time aggregations like storing a rank, percentiles, etc.

By the example of the Eden Network Subgraph, that would enable them to save a lot of database reads and writes:

Instead of (simplified ranking logic from them):

let stakers = network.stakers.map<Staker | null>(id => Staker.load(id)).filter(staker => staker != null);

let sortedStakers = stakers((a, b) => 
    b.staked.div(WEI).minus(a.staked.div(WEI)).toI32()
);

sortedStakers.forEach((staker, index) => {
    staker.rank = BigInt.fromI32(index);
    staker.save();
});

They could do something like:

let stakers = Stakers.findMany({ orderBy: { staked: 'desc' } });

let rankedStakers = sortedStakers.map((staker, index) => {
    staker.rank = BigInt.fromI32(index);
    return staker;
});

Stakers.saveMany(rankedStakers);

Currently, they have ~7000 stakers. That addition would save them 14k-2 database operations.

Current behaviour

In order to update multiple entities in one mapping, subgraph developers are forced to store the ids of all entities that they would like to update in an array on a global singleton entity. That leads to unnecessary big arrays in the database and unnecessary amount of reads and writes.

Affected subgraphs

  • Eden Network
  • Superfluid
  • ...

Inspiration

I personally think the Prisma V2 ORM is pretty well designed: https://www.prisma.io/docs/reference/api-reference/prisma-client-reference#findmany

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the injected entity API described in the proposal and compare its intended findMany and saveMany behavior with the Prisma V2 findMany reference. Define the bulk read, ordering, and write semantics, then identify the relevant implementation and test entry points; done means subgraphs can replace repeated entity loads and saves with these operations.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, typescript
Domain
api, backend-api-design, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.