evstack / evstack/atlas

feat: pipelined fetch/write for indexer

Open
#55 0 comments 0 reactions 1 assignee Claimed by @pthmas View on GitHub
Dominant language
Rust
Stars
2
Forks
1
PR merge metrics
No merged PRs in 30d

Description

## Summary

Currently the indexer fetches a batch of blocks, then writes them to the database, then fetches the next batch. The RPC is idle during writes and the DB is idle during fetches. Pipelining would overlap these operations: fetch batch N+1 while writing batch N.

## Motivation

This is the single biggest throughput improvement available for indexing speed. Based on analysis of [tidx](https://github.com/tempoxyz/tidx), which implements this pattern, estimated **30-50% faster sync** during both initial sync and steady-state head tracking.

## Design

1. After fetching batch N, spawn the DB write as a background task
2. Immediately begin fetching batch N+1
3. Before writing batch N+1, await completion of the batch N write
4. Errors in either fetch or write should still propagate correctly

Key considerations:
- The reorder buffer (`BTreeMap`) already collects results in sequence — pipelining fits naturally after the reorder step
- The binary COPY connection is exclusive during writes, so only one write can be in-flight at a time
- SSE publishing (via `HeadTracker`) currently happens before the DB write, so pipelining doesn't affect SSE timing

## References

- tidx implementation: `sync/engine.rs` — `tick_realtime` fetches batch N+1 while writing batch N

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.