flashbots / flashbots/mempool-dumpster

Timestamp Sorting for Faster Data Access

Open
#65 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
259
Forks
42
PR merge metrics
No merged PRs in 30d

Description

## Current Problem
When we fetch mempool data (transactions), we don't sort transactions by timestamp. This makes it slow to find transactions within specific time ranges.

## Why This Matters
- Searching for transactions in a time range is much slower than it could be
- Each search has to scan through unsorted data
- This affects performance of time-based queries

## Simple Solution
Sort the data by timestamp when saving it. This will make time-based searches much faster. Also we could split it with smaller equal row_group_size like 5000

## Expected Benefits
- Much faster time-range searches

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by locating the code that saves mempool transactions to Parquet and CSV, then trace how timestamps and row_group_size are handled. Determine how sorted data is queried for time ranges and validate that saving in timestamp order with the proposed row-group size improves those searches without breaking existing output.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
data-engineering, performance
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.