[Feature]: Partition-Based Event Processing for Enhanced Subgraph Performance

Beginne damit, Graph Nodes bestehenden Pfad zur Ereignisverarbeitung und die Handhabung von Subgraph-Manifesten zu lesen, mit besonderem Augenmerk darauf, wie Mapping-Ereignishandler eingeplant werden und wie die Reihenfolge erzwungen wird. Definiere die erforderlichen Manifest- und Handler-Schnittstellen, das Verhalten hinsichtlich Nebenläufigkeit und Entitätskonsistenz sowie Tests, die eine geordnete Verarbeitung innerhalb einer Partition und deterministische Ergebnisse über Partitionen hinweg belegen.

Description

Summary

Add support for partition-based parallel event processing in Graph Node, allowing events from the same contract to be processed in parallel based on a developer-defined partition key, while maintaining sequential ordering for events with the same partition key.

Motivation

Currently, Graph Node processes events from a single contract sequentially, even when those events are logically independent. This creates significant performance bottlenecks for singleton contracts that manage multiple independent entities - a pattern becoming increasingly common in modern protocols.

The shift from factory patterns to singleton architectures (like Uniswap V4's PoolManager) exacerbates this issue. Where Uniswap V3 could parallelize across different pool contracts, V4 forces sequential processing of all events from all pools because they originate from a single address.

Real-World Example: Uniswap V4

Uniswap V4 uses a singleton PoolManager contract that manages all pools. Every swap, liquidity change, and pool creation flows through this single contract:

eventHandlers:
  - event: Initialize(indexed bytes32,indexed address,indexed address,uint24,int24,address,uint160,int24)
    handler: handleInitialize
  - event: ModifyLiquidity(indexed bytes32,indexed address,int24,int24,int256,bytes32)
    handler: handleModifyLiquidity
  - event: Swap(indexed bytes32,indexed address,int128,int128,uint160,uint128,int24,uint24)
    handler: handleSwap

The first parameter in each event is the PoolId (bytes32). Events for different pools are completely independent - a swap in the ETH/USDC pool has no relationship to a swap in the WBTC/DAI pool. Yet Graph Node must process them sequentially, creating a severe bottleneck as V4 adoption grows.

A single Ethereum block might contain hundreds of swaps across dozens of pools, all waiting in line for sequential processing.

Proposed Solution

Core Feature

Allow subgraph developers to specify a partition key function in their mappings. Events with different partition keys can be processed in parallel, while events with the same partition key maintain sequential ordering.

Configuration Approach

Add partition configuration to the subgraph manifest:

dataSources:
  - kind: ethereum/contract
    name: PoolManager
    source:
      address: "0x8C4BcBE6b9eF47855f97E675296FA3F6fafa5F1A"
      abi: PoolManager
    mapping:
      kind: ethereum/events
      # NEW: Partition configuration
      partitioning:
        enabled: true
        maxConcurrency: 10  # Optional: limit concurrent partitions
      eventHandlers:
        - event: Swap(indexed bytes32,indexed address,int128,int128,uint160,uint128,int24,uint24)
          handler: handleSwap
          # NEW: Partition by the first parameter (PoolId)
          partitionBy: event.params.id
        - event: ModifyLiquidity(indexed bytes32,indexed address,int24,int24,int256,bytes32)
          handler: handleModifyLiquidity
          partitionBy: event.params.id

Alternative: Define partition key in the handler code:

export function handleSwap(event: Swap): void {
  // Declare partition key at the start of handler
  setPartitionKey(event.params.id.toHexString())
  
  // Rest of handler logic remains unchanged
  let pool = Pool.load(event.params.id)
  // ...
}

Key Requirements

1. Event Ordering Within Partitions

   • Events with the same partition key MUST be processed in block order
   • Maintains data integrity for entity updates

2. Cross-Partition Independence

   • Events with different partition keys can process in parallel
   • No shared state between partitions during event processing

3. Backwards Compatibility

   • Feature is opt-in with zero breaking changes
   • Existing subgraphs continue working exactly as before

4. Deterministic Results

   • Parallel processing must produce identical results to sequential processing
   • Same inputs always produce same indexed state

Implementation Considerations

Entity Store Consistency

The entity store must handle concurrent updates safely:

• Entities with different IDs can be updated in parallel
• Global aggregations may need atomic operations or partition-aware accumulation
• Consider read-write locks per entity ID

Handler Guidelines

Developers must ensure handlers are partition-safe:

• Avoid global variables that could race
• Entity updates should only affect entities within the partition scope
• Cross-partition queries should be read-only during event processing

Performance Monitoring

Expose metrics for:

• Events processed per second by partition
• Partition distribution (to identify hot partitions)
• Queue depths per partition
• Lock contention statistics

Benefits

1. Massive Performance Improvements

• Linear scalability with number of independent entities

2. Better Resource Utilization

• Utilize multiple CPU cores effectively
• Reduce indexing lag during high activity periods

3. Improved Developer Experience

• Simple configuration without restructuring handler logic
• Maintains the same mental model for entity updates

4. Future-Proof Architecture

• Supports the trend toward singleton patterns
• Scales with blockchain activity growth

Use Cases

DeFi Protocols

• Uniswap V4: Partition by PoolId
• Lending Protocols: Partition by market/asset
• Vault Managers: Partition by vault address

NFT/Gaming

• Marketplace Singletons: Partition by collection
• Game State Managers: Partition by game ID or player

Cross-Chain Bridges

• Message Processors: Partition by chain ID or message ID

Generic Registries

• ENS-style Systems: Partition by domain/namespace
• Token Registries: Partition by token address

Example: Uniswap V4 with Partitioning

dataSources:
  - kind: ethereum/contract
    name: PoolManager
    network: mainnet
    source:
      address: "0x..."
      abi: PoolManager
    mapping:
      kind: ethereum/events
      partitioning:
        enabled: true
        maxConcurrency: 20
      eventHandlers:
        - event: Initialize(indexed bytes32,indexed address,indexed address,uint24,int24,address,uint160,int24)
          handler: handleInitialize
          partitionBy: event.params.id
        - event: ModifyLiquidity(indexed bytes32,indexed address,int24,int24,int256,bytes32)
          handler: handleModifyLiquidity
          partitionBy: event.params.id
        - event: Swap(indexed bytes32,indexed address,int128,int128,uint160,uint128,int24,uint24)
          handler: handleSwap
          partitionBy: event.params.id
        - event: Donate(indexed bytes32,indexed address,uint256,uint256)
          handler: handleDonate
          partitionBy: event.params.id

No changes needed to handler implementations - just configuration enables parallel processing!

Technical Considerations

Partition Key Requirements

• Must be deterministic from event data
• Should distribute events evenly to avoid hot partitions
• Typically corresponds to the primary entity ID being updated

Edge Cases

• New entity creation can safely happen in any partition
• Global statistics may need special handling (atomic counters or post-processing aggregation)
• Time-based calculations within a partition remain correct due to ordered processing

Are you aware of any blockers that must be resolved before implementing this feature? If so, which? Link to any relevant GitHub issues.

No response

Some information to help us out

• Tick this box if you plan on implementing this feature yourself.
• I have searched the issue tracker to make sure this issue is not a duplicate.