graphprotocol / graphprotocol/graph-node
Speed up reverts
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 3.2k
- Forks
- 1.1k
- Avg merge
- 4d 1h
- Merged PRs (30d)
- 1
Description
For some subgraphs, reverting blocks is still slow. The best way to speed this up might be to restrict the queries we run to revert the block by the vid of the entity versions that are actually affected by the revert. To facilitate that, graph-node should keep a list of the vid's of entities by block in memory as it moves forward and processes blocks. That list can then be used to speed up reverts.
Even if we only keep this data for a small number of blocks (say 5), it should help in speeding up reverts already. It's ok if we do not have that data for a revert (e.g., after a cold start), we can just fall back to the current behavior. The amount of data to keep should be relatively small, as mappings typically only alter a small number of entities for each block, but we might want to limit this by only keeping the data if there are fewer than N vid's to keep for a block.
Before implementing this, we should analyze the performance of the current queries and compare it to the performance of queries including the vid of the entities affected by the rollback. That should also inform the shape of the data we keep in memory, but will likely look like this for the different operations in a specific block:
create: remember thevidof the new entityupdate: remember thevidof the old and the new version of the entitydelete: remember thevidof the deleted entity version
During a revert, we'd then use this information to narrow down which rows in a table to change, for example the query to delete entity versions that are now in the future would become
delete from things where vid in ($vids)
where vids contains what we recorded as new versions for a create or update
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by profiling graph-node's current block-revert queries and comparing them with queries narrowed by affected entity version IDs (vid). Use those measurements to determine the in-memory per-block data shape and retention limit; done means a measured revert-speed improvement with the existing behavior available when no data is present.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust, sql
- Domain
- backend, database, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100